{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/bridging-the-gap-between-object-and-image","title":"Bridging the Gap between Object and Image-level Representations for Open-Vocabulary Detection","arxiv_id":"2207.03482","date":"2022-07-07","proceeding":null,"authors":["Hanoona Rasheed","Muhammad Maaz","Muhammad Uzair Khattak","Salman Khan","Fahad Shahbaz Khan"],"abstract":"Existing open-vocabulary object detectors typically enlarge their vocabulary sizes by leveraging different forms of weak supervision. This helps generalize to novel objects at inference. Two popular forms of weak-supervision used in open-vocabulary detection (OVD) include pretrained CLIP model and image-level supervision. We note that both these modes of supervision are not optimally aligned for the detection task: CLIP is trained with image-text pairs and lacks precise localization of objects while the image-level supervision has been used with heuristics that do not accurately specify local object regions. In this work, we propose to address this problem by performing object-centric alignment of the language embeddings from the CLIP model. Furthermore, we visually ground the objects with only image-level supervision using a pseudo-labeling process that provides high-quality object proposals and helps expand the vocabulary during training. We establish a bridge between the above two object-alignment strategies via a novel weight transfer function that aggregates their complimentary strengths. In essence, the proposed model seeks to minimize the gap between object and image-centric representations in the OVD setting. On the COCO benchmark, our proposed approach achieves 36.6 AP50 on novel classes, an absolute 8.2 gain over the previous best performance. For LVIS, we surpass the state-of-the-art ViLD model by 5.0 mask AP for rare categories and 3.4 overall. Code: https://github.com/hanoonaR/object-centric-ovd.","url_abs":"https://arxiv.org/abs/2207.03482v3","url_pdf":"https://arxiv.org/pdf/2207.03482v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"bridging-the-gap-between-object-and-image","repo_url":"https://github.com/mmaaz60/mvits_for_class_agnostic_od","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"object","task_name":"Object"},{"task_slug":"open-vocabulary-attribute-detection","task_name":"Open Vocabulary Attribute Detection"},{"task_slug":"open-vocabulary-object-detection","task_name":"Open Vocabulary Object Detection"},{"task_slug":"zero-shot-object-detection","task_name":"Zero-Shot Object Detection"}],"methods":[{"method_slug":"clip","method_name":"CLIP"},{"method_slug":"mavl","method_name":"MAVL"}],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/open-vocabulary-attribute-detection-on-ovad","task":"Open Vocabulary Attribute Detection","dataset":"OVAD benchmark","model":"Object-Centric-OVD (ResNet50)","rank_in_archive_order":5,"of":5,"metrics":{"mean average precision":"14.6"},"uses_additional_data":false},{"leaderboard":"/sota/open-vocabulary-object-detection-on-lvis-v1-0","task":"Open Vocabulary Object Detection","dataset":"LVIS v1.0","model":"Object-Centric-OVD","rank_in_archive_order":23,"of":28,"metrics":{"AP novel-LVIS base training":"21.1"},"uses_additional_data":true},{"leaderboard":"/sota/open-vocabulary-object-detection-on-mscoco","task":"Open Vocabulary Object Detection","dataset":"MSCOCO","model":"Object-Centric-OVD","rank_in_archive_order":16,"of":32,"metrics":{"AP 0.5":"36.9"},"uses_additional_data":false},{"leaderboard":"/sota/open-vocabulary-object-detection-on-1","task":"Open Vocabulary Object Detection","dataset":"Objects365","model":"Object-Centric-OVD","rank_in_archive_order":1,"of":2,"metrics":{"mask AP50":"22.3"},"uses_additional_data":false},{"leaderboard":"/sota/open-vocabulary-object-detection-on","task":"Open Vocabulary Object Detection","dataset":"OpenImages-v4","model":"Object-Centric-OVD","rank_in_archive_order":1,"of":2,"metrics":{"mask AP50":"42.9"},"uses_additional_data":false},{"leaderboard":"/sota/zero-shot-object-detection-on-mscoco","task":"Zero-Shot Object Detection","dataset":"MSCOCO","model":"Object-Centric-OVD","rank_in_archive_order":7,"of":7,"metrics":{"AP":"40.5"},"uses_additional_data":false}],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2207.03482","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2207.03482"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/hanoonaR/object-centric-ovd","reach":{"status":"ok","spdx":"Apache-2.0"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/mmaaz60/mvits_for_class_agnostic_od","reach":null}],"summary":{"ran":2,"unverified":2},"by_repo_kind":{"listed":{"samples":1,"ran":1,"repositories":1},"found_in_text":{"samples":3,"ran":1,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"74d9bf63bb0a32b7","entry":"MDefDETR","repo":"mmaaz60/mvits_for_class_agnostic_od","repo_kind":"listed","path":"models/mdef_detr/mdef_detr.py","file_url":"https://github.com/mmaaz60/mvits_for_class_agnostic_od/blob/HEAD/models/mdef_detr/mdef_detr.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"74d9bf63bb0a32b7"}},{"code_sha256_prefix":"bb583e07209fe58b","entry":"get_fed_loss_inds","repo":"hanoonaR/object-centric-ovd","repo_kind":"found_in_text","path":"ovd/modeling/utils.py","file_url":"https://github.com/hanoonaR/object-centric-ovd/blob/HEAD/ovd/modeling/utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"bb583e07209fe58b"}},{"code_sha256_prefix":"3ccfc563c6913087","entry":"get_frequency_distr","repo":"hanoonaR/object-centric-ovd","repo_kind":"found_in_text","path":"ovd/evaluation/custom_coco_eval.py","file_url":"https://github.com/hanoonaR/object-centric-ovd/blob/HEAD/ovd/evaluation/custom_coco_eval.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"3ccfc563c6913087"}},{"code_sha256_prefix":"5656dece1a24175b","entry":"load_class_freq","repo":"hanoonaR/object-centric-ovd","repo_kind":"found_in_text","path":"ovd/modeling/utils.py","file_url":"https://github.com/hanoonaR/object-centric-ovd/blob/HEAD/ovd/modeling/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"5656dece1a24175b"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}