{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/aligning-and-prompting-everything-all-at-once","title":"Aligning and Prompting Everything All at Once for Universal Visual Perception","arxiv_id":"2312.02153","date":"2023-12-04","proceeding":"CVPR 2024 1","authors":["Yunhang Shen","Chaoyou Fu","Peixian Chen","Mengdan Zhang","Ke Li","Xing Sun","Yunsheng Wu","Shaohui Lin","Rongrong Ji"],"abstract":"Vision foundation models have been explored recently to build general-purpose vision systems. However, predominant paradigms, driven by casting instance-level tasks as an object-word alignment, bring heavy cross-modality interaction, which is not effective in prompting object detection and visual grounding. Another line of work that focuses on pixel-level tasks often encounters a large annotation gap of things and stuff, and suffers from mutual interference between foreground-object and background-class segmentation. In stark contrast to the prevailing methods, we present APE, a universal visual perception model for aligning and prompting everything all at once in an image to perform diverse tasks, i.e., detection, segmentation, and grounding, as an instance-level sentence-object matching paradigm. Specifically, APE advances the convergence of detection and grounding by reformulating language-guided grounding as open-vocabulary detection, which efficiently scales up model prompting to thousands of category vocabularies and region descriptions while maintaining the effectiveness of cross-modality fusion. To bridge the granularity gap of different pixel-level tasks, APE equalizes semantic and panoptic segmentation to proxy instance learning by considering any isolated regions as individual instances. APE aligns vision and language representation on broad data with natural and challenging characteristics all at once without task-specific fine-tuning. The extensive experiments on over 160 datasets demonstrate that, with only one-suit of weights, APE outperforms (or is on par with) the state-of-the-art models, proving that an effective yet universal perception for anything aligning and prompting is indeed feasible. Codes and trained models are released at https://github.com/shenyunhang/APE.","url_abs":"https://arxiv.org/abs/2312.02153v1","url_pdf":"https://arxiv.org/pdf/2312.02153v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"aligning-and-prompting-everything-all-at-once","repo_url":"https://github.com/shenyunhang/ape","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}},{"paper_slug":"aligning-and-prompting-everything-all-at-once","repo_url":"https://github.com/earth-insights/ClassTrans","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"all","task_name":"All"},{"task_slug":"object","task_name":"Object"},{"task_slug":"object-detection","task_name":"Object Detection"},{"task_slug":"panoptic-segmentation","task_name":"Panoptic Segmentation"},{"task_slug":"segmentation","task_name":"Segmentation"},{"task_slug":"sentence","task_name":"Sentence"},{"task_slug":"visual-grounding","task_name":"Visual Grounding"},{"task_slug":"word-alignment","task_name":"Word Alignment"},{"task_slug":"object-detection-1","task_name":"object-detection"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2312.02153","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2312.02153"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/earth-insights/ClassTrans","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/shenyunhang/ape","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran":1,"ran_fixture":1,"unverified":12},"by_repo_kind":{"official":{"samples":5,"ran":2,"repositories":1},"listed":{"samples":9,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"0de113f0340ffcd9","entry":"get_checkpoint_url","repo":"shenyunhang/ape","repo_kind":"official","path":"ape/model_zoo/model_zoo.py","file_url":"https://github.com/shenyunhang/ape/blob/HEAD/ape/model_zoo/model_zoo.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"0de113f0340ffcd9"}},{"code_sha256_prefix":"be924e54ed79731b","entry":"multi_scale_deformable_attn_pytorch","repo":"shenyunhang/ape","repo_kind":"official","path":"ape/layers/multi_scale_deform_attn.py","file_url":"https://github.com/shenyunhang/ape/blob/HEAD/ape/layers/multi_scale_deform_attn.py","link_basis":"harvester_set","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"be924e54ed79731b"}},{"code_sha256_prefix":"47b44963c80f19d1","entry":"create_dummy_class","repo":"shenyunhang/ape","repo_kind":"official","path":"ape/layers/multi_scale_deform_attn.py","file_url":"https://github.com/shenyunhang/ape/blob/HEAD/ape/layers/multi_scale_deform_attn.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"47b44963c80f19d1"}},{"code_sha256_prefix":"5f2ed565ef0da8fc","entry":"create_dummy_func","repo":"shenyunhang/ape","repo_kind":"official","path":"ape/layers/multi_scale_deform_attn.py","file_url":"https://github.com/shenyunhang/ape/blob/HEAD/ape/layers/multi_scale_deform_attn.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"5f2ed565ef0da8fc"}},{"code_sha256_prefix":"e8d6610aacdd981b","entry":"get_config_file","repo":"shenyunhang/ape","repo_kind":"official","path":"ape/model_zoo/model_zoo.py","file_url":"https://github.com/shenyunhang/ape/blob/HEAD/ape/model_zoo/model_zoo.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"e8d6610aacdd981b"}},{"code_sha256_prefix":"25190d960f6a2e87","entry":"get_image_and_label","repo":"earth-insights/ClassTrans","repo_kind":"listed","path":"src/dataset/data.py","file_url":"https://github.com/earth-insights/ClassTrans/blob/HEAD/src/dataset/data.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"25190d960f6a2e87"}},{"code_sha256_prefix":"807451ec22e46bdb","entry":"get_next_run_id","repo":"earth-insights/ClassTrans","repo_kind":"listed","path":"src/utils.py","file_url":"https://github.com/earth-insights/ClassTrans/blob/HEAD/src/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"807451ec22e46bdb"}},{"code_sha256_prefix":"bd491478bac24e5a","entry":"get_val_loader","repo":"earth-insights/ClassTrans","repo_kind":"listed","path":"src/dataset/data.py","file_url":"https://github.com/earth-insights/ClassTrans/blob/HEAD/src/dataset/data.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"bd491478bac24e5a"}},{"code_sha256_prefix":"25fea6c1ac82a8c5","entry":"is_image_file","repo":"earth-insights/ClassTrans","repo_kind":"listed","path":"src/dataset/utils.py","file_url":"https://github.com/earth-insights/ClassTrans/blob/HEAD/src/dataset/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"25fea6c1ac82a8c5"}},{"code_sha256_prefix":"f54083dba77e2917","entry":"label2rgb","repo":"earth-insights/ClassTrans","repo_kind":"listed","path":"src/visualize.py","file_url":"https://github.com/earth-insights/ClassTrans/blob/HEAD/src/visualize.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f54083dba77e2917"}},{"code_sha256_prefix":"3baca7953e57a8e0","entry":"main_process","repo":"earth-insights/ClassTrans","repo_kind":"listed","path":"src/utils.py","file_url":"https://github.com/earth-insights/ClassTrans/blob/HEAD/src/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"3baca7953e57a8e0"}},{"code_sha256_prefix":"4a14363d223db042","entry":"make_dataset","repo":"earth-insights/ClassTrans","repo_kind":"listed","path":"src/dataset/utils.py","file_url":"https://github.com/earth-insights/ClassTrans/blob/HEAD/src/dataset/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"4a14363d223db042"}},{"code_sha256_prefix":"cc9e1a1081e7703d","entry":"map_","repo":"earth-insights/ClassTrans","repo_kind":"listed","path":"src/utils.py","file_url":"https://github.com/earth-insights/ClassTrans/blob/HEAD/src/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"cc9e1a1081e7703d"}},{"code_sha256_prefix":"09e34c3bca52dcda","entry":"process_image","repo":"earth-insights/ClassTrans","repo_kind":"listed","path":"src/dataset/utils.py","file_url":"https://github.com/earth-insights/ClassTrans/blob/HEAD/src/dataset/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"09e34c3bca52dcda"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}