{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/pem-prototype-based-efficient-maskformer-for","title":"PEM: Prototype-based Efficient MaskFormer for Image Segmentation","arxiv_id":"2402.19422","date":"2024-02-29","proceeding":"CVPR 2024 1","authors":["Niccolò Cavagnero","Gabriele Rosi","Claudia Cuttano","Francesca Pistilli","Marco Ciccone","Giuseppe Averta","Fabio Cermelli"],"abstract":"Recent transformer-based architectures have shown impressive results in the field of image segmentation. Thanks to their flexibility, they obtain outstanding performance in multiple segmentation tasks, such as semantic and panoptic, under a single unified framework. To achieve such impressive performance, these architectures employ intensive operations and require substantial computational resources, which are often not available, especially on edge devices. To fill this gap, we propose Prototype-based Efficient MaskFormer (PEM), an efficient transformer-based architecture that can operate in multiple segmentation tasks. PEM proposes a novel prototype-based cross-attention which leverages the redundancy of visual features to restrict the computation and improve the efficiency without harming the performance. In addition, PEM introduces an efficient multi-scale feature pyramid network, capable of extracting features that have high semantic content in an efficient way, thanks to the combination of deformable convolutions and context-based self-modulation. We benchmark the proposed PEM architecture on two tasks, semantic and panoptic segmentation, evaluated on two different datasets, Cityscapes and ADE20K. PEM demonstrates outstanding performance on every task and dataset, outperforming task-specific architectures while being comparable and even better than computationally-expensive baselines.","url_abs":"https://arxiv.org/abs/2402.19422v3","url_pdf":"https://arxiv.org/pdf/2402.19422v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"pem-prototype-based-efficient-maskformer-for","repo_url":"https://github.com/niccolocavagnero/pem","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":"image-segmentation","task_name":"Image Segmentation"},{"task_slug":"panoptic-segmentation","task_name":"Panoptic Segmentation"},{"task_slug":"segmentation","task_name":"Segmentation"},{"task_slug":"semantic-segmentation","task_name":"Semantic Segmentation"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2402.19422","atlas_url":"https://app.syntology.ai/?focus=2402.19422","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2402.19422"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/niccolocavagnero/pem","reach":{"status":"ok"}},{"provenance":"deterministic:regex_extraction","url":"https://github.com/NiccoloCavagnero/PEM","reach":{"status":"ok"}}],"summary":{"ran":2,"unverified":6},"by_repo_kind":{"official":{"samples":8,"ran":2,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":8,"samples":[{"code_sha256_prefix":"28baf4547c97b344","entry":"LocalRepresentation","repo":"NiccoloCavagnero/PEM","repo_kind":"official","path":"pem/modeling/transformer_decoder/pem_transformer_decoder.py","file_url":"https://github.com/NiccoloCavagnero/PEM/blob/HEAD/pem/modeling/transformer_decoder/pem_transformer_decoder.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"28baf4547c97b344"}},{"code_sha256_prefix":"260da9226b359802","entry":"PEM_CA","repo":"NiccoloCavagnero/PEM","repo_kind":"official","path":"pem/modeling/transformer_decoder/pem_transformer_decoder.py","file_url":"https://github.com/NiccoloCavagnero/PEM/blob/HEAD/pem/modeling/transformer_decoder/pem_transformer_decoder.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"260da9226b359802"}},{"code_sha256_prefix":"bc2cb481a75c370d","entry":"batch_dice_loss","repo":"niccolocavagnero/pem","repo_kind":"official","path":"pem/modeling/matcher.py","file_url":"https://github.com/niccolocavagnero/pem/blob/HEAD/pem/modeling/matcher.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"bc2cb481a75c370d"}},{"code_sha256_prefix":"1edd24985036b0bf","entry":"batch_sigmoid_ce_loss","repo":"niccolocavagnero/pem","repo_kind":"official","path":"pem/modeling/matcher.py","file_url":"https://github.com/niccolocavagnero/pem/blob/HEAD/pem/modeling/matcher.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"1edd24985036b0bf"}},{"code_sha256_prefix":"2dcb8123d89bb1ff","entry":"calculate_uncertainty","repo":"niccolocavagnero/pem","repo_kind":"official","path":"pem/modeling/criterion.py","file_url":"https://github.com/niccolocavagnero/pem/blob/HEAD/pem/modeling/criterion.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"2dcb8123d89bb1ff"}},{"code_sha256_prefix":"89f75e54ff128be0","entry":"dice_loss","repo":"niccolocavagnero/pem","repo_kind":"official","path":"pem/modeling/criterion.py","file_url":"https://github.com/niccolocavagnero/pem/blob/HEAD/pem/modeling/criterion.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"89f75e54ff128be0"}},{"code_sha256_prefix":"58cc9ff3bf75e753","entry":"nested_tensor_from_tensor_list","repo":"niccolocavagnero/pem","repo_kind":"official","path":"pem/utils/misc.py","file_url":"https://github.com/niccolocavagnero/pem/blob/HEAD/pem/utils/misc.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"58cc9ff3bf75e753"}},{"code_sha256_prefix":"d0c61e8dba511aa3","entry":"sigmoid_ce_loss","repo":"niccolocavagnero/pem","repo_kind":"official","path":"pem/modeling/criterion.py","file_url":"https://github.com/niccolocavagnero/pem/blob/HEAD/pem/modeling/criterion.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"d0c61e8dba511aa3"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}