{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/perception-encoder-the-best-visual-embeddings","title":"Perception Encoder: The best visual embeddings are not at the output of the network","arxiv_id":"2504.13181","date":"2025-04-17","proceeding":null,"authors":["Daniel Bolya","Po-Yao Huang","Peize Sun","Jang Hyun Cho","Andrea Madotto","Chen Wei","Tengyu Ma","Jiale Zhi","Jathushan Rajasegaran","Hanoona Rasheed","Junke Wang","Marco Monteiro","Hu Xu","Shiyu Dong","Nikhila Ravi","Daniel Li","Piotr Dollár","Christoph Feichtenhofer"],"abstract":"We introduce Perception Encoder (PE), a state-of-the-art encoder for image and video understanding trained via simple vision-language learning. Traditionally, vision encoders have relied on a variety of pretraining objectives, each tailored to specific downstream tasks such as classification, captioning, or localization. Surprisingly, after scaling our carefully tuned image pretraining recipe and refining with our robust video data engine, we find that contrastive vision-language training alone can produce strong, general embeddings for all of these downstream tasks. There is only one caveat: these embeddings are hidden within the intermediate layers of the network. To draw them out, we introduce two alignment methods, language alignment for multimodal language modeling, and spatial alignment for dense prediction. Together with the core contrastive checkpoint, our PE family of models achieves state-of-the-art performance on a wide variety of tasks, including zero-shot image and video classification and retrieval; document, image, and video Q&A; and spatial tasks such as detection, depth estimation, and tracking. To foster further research, we are releasing our models, code, and a novel dataset of synthetically and human-annotated videos.","url_abs":"https://arxiv.org/abs/2504.13181v1","url_pdf":"https://arxiv.org/pdf/2504.13181v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"perception-encoder-the-best-visual-embeddings","repo_url":"https://github.com/facebookresearch/perception_models","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"depth-estimation","task_name":"Depth Estimation"},{"task_slug":"language-modeling","task_name":"Language Modeling"},{"task_slug":"language-modelling","task_name":"Language Modelling"},{"task_slug":"object-detection","task_name":"Object Detection"},{"task_slug":"video-classification","task_name":"Video Classification"},{"task_slug":"video-understanding","task_name":"Video Understanding"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/object-detection-on-coco-minival","task":"Object Detection","dataset":"COCO minival","model":"PE_spatial (DETA)","rank_in_archive_order":1,"of":220,"metrics":{"Params (M)":"1900","box AP":"66.0"},"uses_additional_data":true}],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2504.13181","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2504.13181"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/facebookresearch/perception_models","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran_violates":1,"ran_fixture":2,"ran":6,"ran_draft_wrong":3,"unverified":6},"by_repo_kind":{"listed":{"samples":18,"ran":12,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"ac8fe530cdad4d8c","entry":"dice_loss","repo":"facebookresearch/perception_models","repo_kind":"listed","path":"apps/detection/DETA_pe/models/segmentation.py","file_url":"https://github.com/facebookresearch/perception_models/blob/HEAD/apps/detection/DETA_pe/models/segmentation.py","link_basis":"harvester_set","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"ac8fe530cdad4d8c"}},{"code_sha256_prefix":"87577b3ff9d32712","entry":"drop_path","repo":"facebookresearch/perception_models","repo_kind":"listed","path":"apps/detection/DETA_pe/models/swin.py","file_url":"https://github.com/facebookresearch/perception_models/blob/HEAD/apps/detection/DETA_pe/models/swin.py","link_basis":"harvester_set","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"87577b3ff9d32712"}},{"code_sha256_prefix":"bb583e07209fe58b","entry":"get_fed_loss_inds","repo":"facebookresearch/perception_models","repo_kind":"listed","path":"apps/detection/DETA_pe/models/utils_fed_loss.py","file_url":"https://github.com/facebookresearch/perception_models/blob/HEAD/apps/detection/DETA_pe/models/utils_fed_loss.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"bb583e07209fe58b"}},{"code_sha256_prefix":"edb9869f131321d2","entry":"get_rel_pos","repo":"facebookresearch/perception_models","repo_kind":"listed","path":"apps/detection/DETA_pe/models/utils_d2.py","file_url":"https://github.com/facebookresearch/perception_models/blob/HEAD/apps/detection/DETA_pe/models/utils_d2.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"edb9869f131321d2"}},{"code_sha256_prefix":"d28e238107e0aa1b","entry":"nonzero_tuple","repo":"facebookresearch/perception_models","repo_kind":"listed","path":"apps/detection/DETA_pe/models/assigner.py","file_url":"https://github.com/facebookresearch/perception_models/blob/HEAD/apps/detection/DETA_pe/models/assigner.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"d28e238107e0aa1b"}},{"code_sha256_prefix":"58823d9435a8751b","entry":"rotate_half","repo":"facebookresearch/perception_models","repo_kind":"listed","path":"apps/detection/DETA_pe/models/pev1.py","file_url":"https://github.com/facebookresearch/perception_models/blob/HEAD/apps/detection/DETA_pe/models/pev1.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"58823d9435a8751b"}},{"code_sha256_prefix":"5c0711aada67957e","entry":"sigmoid_focal_loss","repo":"facebookresearch/perception_models","repo_kind":"listed","path":"apps/detection/DETA_pe/models/segmentation.py","file_url":"https://github.com/facebookresearch/perception_models/blob/HEAD/apps/detection/DETA_pe/models/segmentation.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"5c0711aada67957e"}},{"code_sha256_prefix":"19a04bf3d341cedb","entry":"subsample_labels","repo":"facebookresearch/perception_models","repo_kind":"listed","path":"apps/detection/DETA_pe/models/assigner.py","file_url":"https://github.com/facebookresearch/perception_models/blob/HEAD/apps/detection/DETA_pe/models/assigner.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"19a04bf3d341cedb"}},{"code_sha256_prefix":"f9fd6241d935f07b","entry":"window_partition","repo":"facebookresearch/perception_models","repo_kind":"listed","path":"apps/detection/DETA_pe/models/swin.py","file_url":"https://github.com/facebookresearch/perception_models/blob/HEAD/apps/detection/DETA_pe/models/swin.py","link_basis":"harvester_set","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"f9fd6241d935f07b"}},{"code_sha256_prefix":"750facb8c52e8864","entry":"window_partition","repo":"facebookresearch/perception_models","repo_kind":"listed","path":"apps/detection/DETA_pe/models/utils_d2.py","file_url":"https://github.com/facebookresearch/perception_models/blob/HEAD/apps/detection/DETA_pe/models/utils_d2.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"750facb8c52e8864"}},{"code_sha256_prefix":"fb32094c6dbece71","entry":"window_reverse","repo":"facebookresearch/perception_models","repo_kind":"listed","path":"apps/detection/DETA_pe/models/swin.py","file_url":"https://github.com/facebookresearch/perception_models/blob/HEAD/apps/detection/DETA_pe/models/swin.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"fb32094c6dbece71"}},{"code_sha256_prefix":"535942f820542fb9","entry":"window_unpartition","repo":"facebookresearch/perception_models","repo_kind":"listed","path":"apps/detection/DETA_pe/models/utils_d2.py","file_url":"https://github.com/facebookresearch/perception_models/blob/HEAD/apps/detection/DETA_pe/models/utils_d2.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"535942f820542fb9"}},{"code_sha256_prefix":"1cf6c2f532b241f1","entry":"batched_soft_nms","repo":"facebookresearch/perception_models","repo_kind":"listed","path":"apps/detection/DETA_pe/models/utils_softnms.py","file_url":"https://github.com/facebookresearch/perception_models/blob/HEAD/apps/detection/DETA_pe/models/utils_softnms.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"1cf6c2f532b241f1"}},{"code_sha256_prefix":"00e3006fbfc5835f","entry":"broadcat","repo":"facebookresearch/perception_models","repo_kind":"listed","path":"apps/detection/DETA_pe/models/pev1.py","file_url":"https://github.com/facebookresearch/perception_models/blob/HEAD/apps/detection/DETA_pe/models/pev1.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"00e3006fbfc5835f"}},{"code_sha256_prefix":"f733619c7c9a0a40","entry":"get_abs_pos","repo":"facebookresearch/perception_models","repo_kind":"listed","path":"apps/detection/DETA_pe/models/pev1.py","file_url":"https://github.com/facebookresearch/perception_models/blob/HEAD/apps/detection/DETA_pe/models/pev1.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"f733619c7c9a0a40"}},{"code_sha256_prefix":"0547030fc587fd16","entry":"load_class_freq","repo":"facebookresearch/perception_models","repo_kind":"listed","path":"apps/detection/DETA_pe/models/utils_fed_loss.py","file_url":"https://github.com/facebookresearch/perception_models/blob/HEAD/apps/detection/DETA_pe/models/utils_fed_loss.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"0547030fc587fd16"}},{"code_sha256_prefix":"626753a2de4f5009","entry":"sample_topk_per_gt","repo":"facebookresearch/perception_models","repo_kind":"listed","path":"apps/detection/DETA_pe/models/assigner.py","file_url":"https://github.com/facebookresearch/perception_models/blob/HEAD/apps/detection/DETA_pe/models/assigner.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"626753a2de4f5009"}},{"code_sha256_prefix":"8b89ea91621de82b","entry":"soft_nms","repo":"facebookresearch/perception_models","repo_kind":"listed","path":"apps/detection/DETA_pe/models/utils_softnms.py","file_url":"https://github.com/facebookresearch/perception_models/blob/HEAD/apps/detection/DETA_pe/models/utils_softnms.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"8b89ea91621de82b"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}