{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/your-vit-is-secretly-an-image-segmentation-1","title":"Your ViT is Secretly an Image Segmentation Model","arxiv_id":"2503.19108","date":"2025-03-24","proceeding":"CVPR 2025 1","authors":["Tommie Kerssies","Niccolò Cavagnero","Alexander Hermans","Narges Norouzi","Giuseppe Averta","Bastian Leibe","Gijs Dubbelman","Daan de Geus"],"abstract":"Vision Transformers (ViTs) have shown remarkable performance and scalability across various computer vision tasks. To apply single-scale ViTs to image segmentation, existing methods adopt a convolutional adapter to generate multi-scale features, a pixel decoder to fuse these features, and a Transformer decoder that uses the fused features to make predictions. In this paper, we show that the inductive biases introduced by these task-specific components can instead be learned by the ViT itself, given sufficiently large models and extensive pre-training. Based on these findings, we introduce the Encoder-only Mask Transformer (EoMT), which repurposes the plain ViT architecture to conduct image segmentation. With large-scale models and pre-training, EoMT obtains a segmentation accuracy similar to state-of-the-art models that use task-specific components. At the same time, EoMT is significantly faster than these methods due to its architectural simplicity, e.g., up to 4x faster with ViT-L. Across a range of model sizes, EoMT demonstrates an optimal balance between segmentation accuracy and prediction speed, suggesting that compute resources are better spent on scaling the ViT itself rather than adding architectural complexity. Code: https://www.tue-mps.org/eomt/.","url_abs":"https://arxiv.org/abs/2503.19108v1","url_pdf":"https://arxiv.org/pdf/2503.19108v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"your-vit-is-secretly-an-image-segmentation-1","repo_url":"https://github.com/tue-mps/eomt","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"decoder","task_name":"Decoder"},{"task_slug":"image-segmentation","task_name":"Image Segmentation"},{"task_slug":"segmentation","task_name":"Segmentation"},{"task_slug":"semantic-segmentation","task_name":"Semantic Segmentation"}],"methods":[{"method_slug":"adopt","method_name":"ADOPT"},{"method_slug":"absolute-position-encodings","method_name":"Absolute Position Encodings"},{"method_slug":"adam","method_name":"Adam"},{"method_slug":"adapter","method_name":"Adapter"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"bpe","method_name":"BPE"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"label-smoothing","method_name":"Label Smoothing"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"position-wise-feed-forward-layer","method_name":"Position-Wise Feed-Forward Layer"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"transformer","method_name":"Transformer"}],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/panoptic-segmentation-on-ade20k-val","task":"Panoptic Segmentation","dataset":"ADE20K val","model":"EoMT (DINOv2-g, single-scale, 1280x1280, COCO pre-trained)","rank_in_archive_order":5,"of":25,"metrics":{"PQ":"52.8"},"uses_additional_data":true},{"leaderboard":"/sota/panoptic-segmentation-on-coco-minival","task":"Panoptic Segmentation","dataset":"COCO minival","model":"EoMT (DINOv2-g, single-scale, 1280x1280)","rank_in_archive_order":6,"of":31,"metrics":{"PQ":"59.2"},"uses_additional_data":false},{"leaderboard":"/sota/semantic-segmentation-on-ade20k","task":"Semantic Segmentation","dataset":"ADE20K","model":"EoMT (DINOv2-L, single-scale, 512x512)","rank_in_archive_order":21,"of":235,"metrics":{"GFLOPs":"721","GFLOPs (512 x 512)":"721","Mean IoU (class)":"58.4","Params (M)":"316","Validation mIoU":"58.4"},"uses_additional_data":false},{"leaderboard":"/sota/semantic-segmentation-on-ade20k-val","task":"Semantic Segmentation","dataset":"ADE20K val","model":"EoMT (DINOv2-L, single-scale, 512x512)","rank_in_archive_order":13,"of":95,"metrics":{"mIoU":"58.4"},"uses_additional_data":false},{"leaderboard":"/sota/semantic-segmentation-on-cityscapes-val","task":"Semantic Segmentation","dataset":"Cityscapes val","model":"EoMT (DINOv2-L, single-scale, 1024x1024)","rank_in_archive_order":20,"of":99,"metrics":{"FPS":"25","Validation mIoU":"84.2","mIoU":"84.2"},"uses_additional_data":false}],"syntology":{"syntology_url":"https://syntology.ai/paper/2503.19108","atlas_url":"https://app.syntology.ai/?focus=2503.19108","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2503.19108"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/tue-mps/eomt","reach":null}],"summary":{"ran":1,"unverified":1},"by_repo_kind":{"official":{"samples":2,"ran":1,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"51c0ff8e7d577ed2","entry":"ScaleBlock","repo":"tue-mps/eomt","repo_kind":"official","path":"models/eomt.py","file_url":"https://github.com/tue-mps/eomt/blob/HEAD/models/eomt.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"51c0ff8e7d577ed2"}},{"code_sha256_prefix":"7ef7ad819fd53cab","entry":"EoMT","repo":"tue-mps/eomt","repo_kind":"official","path":"models/eomt.py","file_url":"https://github.com/tue-mps/eomt/blob/HEAD/models/eomt.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"7ef7ad819fd53cab"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}