Abstract
Comparing transformer backbones for image segmentation is confounded: each is paired with a different decoder, recipe, and pretraining, so reported differences rarely reflect the backbone itself. We introduce the Lightweight Universal Mask Adapter (LUMA), a lightweight, backbone-agnostic mask-transformer head that treats any backbone as a black-box feature extractor, letting a set of queries read from its features through cheap cross-attention. LUMA matches the accuracy of EoMT, the state-of-the-art efficient ViT-segmenter, at cost parity, while attaching unchanged to hierarchical, convolutional, and mixture-of-experts backbones alike. Holding this head fixed, we benchmark 20 backbones, 11 pretraining schemes and a range of resolutions on ADE20K, COCO, and Cityscapes under one modern recipe. We find that “efficient” token mixers fail to deliver efficiency even at the high resolutions that motivate them, with plain ViT holding the throughput Pareto front at every resolution. Additionally, the pretraining objective, not the architecture, the lever the field has tuned hardest, governs segmentation quality.
Full results and training details are in the paper.
Comparing backbones fairly
Which backbone should you use for image segmentation? Published results make that surprisingly hard to answer. Backbones usually come with different decoders, pretraining, and training recipes, so a better score doesn’t tell us which part helped.
EoMT is an efficient segmentation method for plain ViTs, but its queries live inside the backbone’s attention. That makes it unsuitable for comparing backbones with windowed attention, changing feature widths, or entirely different token mixers. We built LUMA to make those comparisons possible.
LUMA: one head for different backbones
The Lightweight Universal Mask Adapter keeps its segmentation queries separate from the backbone. The backbone processes the image normally; the queries read its features through cross-attention.
We tap the last two backbone blocks. At each, the queries read the image features, attend to each other, and pass through the backbone’s own feed-forward layer. They also reuse its width-changing operations when needed. Backbones without a suitable feed-forward layer get a separate query MLP.
Semantic segmentation uses 100 queries and unmasked cross-attention. Panoptic segmentation, which also separates individual object instances, uses 200 queries and masked cross-attention during training. We gradually remove that masking using EoMT’s annealing schedule. Query self-attention helps keep instances apart.
The same design works with plain ViTs, hierarchical models, CNNs, and mixture-of-experts backbones, without modifying their token mixers.
Does LUMA match specialized heads?
With a DINOv2 ViT-L backbone, LUMA reaches 84.6 mIoU on Cityscapes and 58.6 on ADE20K. It matches EoMT’s 58.3 PQ on COCO and 51.7 PQ on ADE20K, at comparable compute cost.
On ADE20K semantic segmentation, LUMA also runs about seven times faster than ViT-Adapter + Mask2Former.
The benchmark
We compare 20 backbones and 11 pretraining schemes across image resolutions, using ADE20K and Cityscapes for semantic segmentation and ADE20K and COCO for panoptic segmentation. We measure mIoU or PQ alongside throughput, inference memory, and FLOPs.
For the architecture comparison, all backbones start from the same DeiT III pretraining recipe on ImageNet-21k. Separate experiments vary the pretraining objective and the balance between model size and resolution. Efficiency measurements use a single H100 with bf16 and torch.compile.
What we found
Plain ViT remains hard to beat
Plain ViT stays on the throughput Pareto front, even without FlashAttention. Enabling it roughly doubles throughput. Sparse attention models and FocalNet come close; NextViT is a good option when memory matters most.
Several mixers struggle as resolution grows: Linformer falls from 38.9 to 4.9 mIoU, and Hydra from 34.9 to 7.5. The speed comparisons also reflect kernel optimization, so better implementations remain a promising route for efficient alternatives.
Pretraining matters more than the token mixer
Dense objectives such as masked image modeling and DINO-style self-distillation transfer well. Supervised DeiT III sits roughly eight mIoU/PQ points below the trend predicted by its classification accuracy. EVA-02 reaches 59.1 mIoU and 49.7 PQ on ADE20K with ViT-L, while DINOv3 leads on Cityscapes and COCO.
ImageNet accuracy is an unreliable guide
Similar classification scores can hide large differences in segmentation quality. When comparing backbones under the same pretraining, ImageNet accuracy is correlated with segmentation performance at $r=0.8$, but with significant outliers.
For ViT models with different pretrainings (plot above), the ImageNet linear-probe ranking tells us even less: on ADE20K, removing the weakest outliers leaves a rank correlation of just $\rho = 0.21$ and for ViT-H models on COCO panoptic segmentation the trend is even negative!
Spend most extra compute on model size
Our DINOv2 scaling experiments suggest a rough 80/20 rule: spend about 80% of added compute on model size and 20% on higher resolution. This assumes the pretraining remains effective as the model grows.
What to use
Start with plain ViT and FlashAttention, initialize from DINOv3 or EVA-02, and put most extra compute into a larger backbone.
Citation
If you use this work, please cite our paper:
@misc{nauen2026luma,
title = {LUMA: Benchmarking Segmentation via a Lightweight Universal Mask
Adapter},
author = {Tobias Christian Nauen and Anosh Billimoria and Federico Raue and
Stanislav Frolov and Brian Moser and Andreas Dengel},
year = {2026},
eprint = {2607.00687},
archiveprefix = {arXiv},
primaryclass = {cs.CV},
note = {Accepted at WACV 2027},
}
