{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/open-vocabulary-detr-with-conditional","title":"Open-Vocabulary DETR with Conditional Matching","arxiv_id":"2203.11876","date":"2022-03-22","proceeding":null,"authors":["Yuhang Zang","Wei Li","Kaiyang Zhou","Chen Huang","Chen Change Loy"],"abstract":"Open-vocabulary object detection, which is concerned with the problem of detecting novel objects guided by natural language, has gained increasing attention from the community. Ideally, we would like to extend an open-vocabulary detector such that it can produce bounding box predictions based on user inputs in form of either natural language or exemplar image. This offers great flexibility and user experience for human-computer interaction. To this end, we propose a novel open-vocabulary detector based on DETR -- hence the name OV-DETR -- which, once trained, can detect any object given its class name or an exemplar image. The biggest challenge of turning DETR into an open-vocabulary detector is that it is impossible to calculate the classification cost matrix of novel classes without access to their labeled images. To overcome this challenge, we formulate the learning objective as a binary matching one between input queries (class name or exemplar image) and the corresponding objects, which learns useful correspondence to generalize to unseen queries during testing. For training, we choose to condition the Transformer decoder on the input embeddings obtained from a pre-trained vision-language model like CLIP, in order to enable matching for both text and image queries. With extensive experiments on LVIS and COCO datasets, we demonstrate that our OV-DETR -- the first end-to-end Transformer-based open-vocabulary detector -- achieves non-trivial improvements over current state of the arts.","url_abs":"https://arxiv.org/abs/2203.11876v2","url_pdf":"https://arxiv.org/pdf/2203.11876v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"open-vocabulary-detr-with-conditional","repo_url":"https://github.com/yuhangzang/ov-detr","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}},{"paper_slug":"open-vocabulary-detr-with-conditional","repo_url":"https://github.com/hchoi256/i-halla-v1.0","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"open-vocabulary-detr-with-conditional","repo_url":"https://github.com/kaist-cvml-lab/part-clipseg","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"open-vocabulary-detr-with-conditional","repo_url":"https://github.com/kaist-cvml/part-clipseg","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"jax","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"language-modelling","task_name":"Language Modelling"},{"task_slug":"object-detection","task_name":"Object Detection"},{"task_slug":"open-vocabulary-object-detection","task_name":"Open Vocabulary Object Detection"},{"task_slug":"open-vocabulary-object-detection","task_name":"Open-vocabulary object detection"},{"task_slug":"object-detection-1","task_name":"object-detection"}],"methods":[{"method_slug":"absolute-position-encodings","method_name":"Absolute Position Encodings"},{"method_slug":"adam","method_name":"Adam"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"bpe","method_name":"BPE"},{"method_slug":"clip","method_name":"CLIP"},{"method_slug":"convolution","method_name":"Convolution"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"detr","method_name":"Detr"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"feedforward-network","method_name":"Feedforward Network"},{"method_slug":"label-smoothing","method_name":"Label Smoothing"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"position-wise-feed-forward-layer","method_name":"Position-Wise Feed-Forward Layer"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"transformer","method_name":"Transformer"}],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/open-vocabulary-object-detection-on-mscoco","task":"Open Vocabulary Object Detection","dataset":"MSCOCO","model":"OV-DERT","rank_in_archive_order":26,"of":32,"metrics":{"AP 0.5":"29.4"},"uses_additional_data":false}],"syntology":{"syntology_url":"https://syntology.ai/paper/2203.11876","atlas_url":"https://app.syntology.ai/?focus=2203.11876","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2203.11876"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/kaist-cvml/part-clipseg","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/yuhangzang/ov-detr","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/hchoi256/i-halla-v1.0","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/kaist-cvml-lab/part-clipseg","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"deterministic:regex_extraction","url":"https://github.com/yuhangzang/OV-DETR","reach":{"status":"ok"}}],"summary":{"ran":5,"ran_draft_wrong":1,"unverified":2},"by_repo_kind":{"official":{"samples":4,"ran":3,"repositories":1},"listed":{"samples":4,"ran":3,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":4,"samples":[{"code_sha256_prefix":"0415bb0bee56bc59","entry":"DeformableDETR","repo":"yuhangzang/OV-DETR","repo_kind":"official","path":"ovdetr/models/model.py","file_url":"https://github.com/yuhangzang/OV-DETR/blob/HEAD/ovdetr/models/model.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"0415bb0bee56bc59"}},{"code_sha256_prefix":"a59cf95e6b922e44","entry":"NestedTensor","repo":"yuhangzang/OV-DETR","repo_kind":"official","path":"ovdetr/models/model.py","file_url":"https://github.com/yuhangzang/OV-DETR/blob/HEAD/ovdetr/models/model.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"a59cf95e6b922e44"}},{"code_sha256_prefix":"2bdfadf4cfb18cd8","entry":"load_image","repo":"hchoi256/i-halla-v1.0","repo_kind":"listed","path":"utils.py","file_url":"https://github.com/hchoi256/i-halla-v1.0/blob/HEAD/utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"2bdfadf4cfb18cd8"}},{"code_sha256_prefix":"57b6cc60d20c5e85","entry":"nested_tensor_from_tensor_list","repo":"yuhangzang/OV-DETR","repo_kind":"official","path":"ovdetr/models/model.py","file_url":"https://github.com/yuhangzang/OV-DETR/blob/HEAD/ovdetr/models/model.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"57b6cc60d20c5e85"}},{"code_sha256_prefix":"69eda7e624ba1809","entry":"parse_questions","repo":"hchoi256/i-halla-v1.0","repo_kind":"listed","path":"agents/CoIAgent.py","file_url":"https://github.com/hchoi256/i-halla-v1.0/blob/HEAD/agents/CoIAgent.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"69eda7e624ba1809"}},{"code_sha256_prefix":"75d5bef2a1de5b6b","entry":"parse_questions","repo":"hchoi256/i-halla-v1.0","repo_kind":"listed","path":"agents/QAAgent.py","file_url":"https://github.com/hchoi256/i-halla-v1.0/blob/HEAD/agents/QAAgent.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"75d5bef2a1de5b6b"}},{"code_sha256_prefix":"e82eea2c13d31778","entry":"OVDETR","repo":"yuhangzang/OV-DETR","repo_kind":"official","path":"ovdetr/models/model.py","file_url":"https://github.com/yuhangzang/OV-DETR/blob/HEAD/ovdetr/models/model.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"e82eea2c13d31778"}},{"code_sha256_prefix":"82658872a11788ab","entry":"load_text","repo":"hchoi256/i-halla-v1.0","repo_kind":"listed","path":"utils.py","file_url":"https://github.com/hchoi256/i-halla-v1.0/blob/HEAD/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"82658872a11788ab"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}