{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/ovmr-open-vocabulary-recognition-with-multi","title":"OVMR: Open-Vocabulary Recognition with Multi-Modal References","arxiv_id":"2406.04675","date":"2024-06-07","proceeding":"CVPR 2024 1","authors":["Zehong Ma","Shiliang Zhang","Longhui Wei","Qi Tian"],"abstract":"The challenge of open-vocabulary recognition lies in the model has no clue of new categories it is applied to. Existing works have proposed different methods to embed category cues into the model, \\eg, through few-shot fine-tuning, providing category names or textual descriptions to Vision-Language Models. Fine-tuning is time-consuming and degrades the generalization capability. Textual descriptions could be ambiguous and fail to depict visual details. This paper tackles open-vocabulary recognition from a different perspective by referring to multi-modal clues composed of textual descriptions and exemplar images. Our method, named OVMR, adopts two innovative components to pursue a more robust category cues embedding. A multi-modal classifier is first generated by dynamically complementing textual descriptions with image exemplars. A preference-based refinement module is hence applied to fuse uni-modal and multi-modal classifiers, with the aim to alleviate issues of low-quality exemplar images or textual descriptions. The proposed OVMR is a plug-and-play module, and works well with exemplar images randomly crawled from the Internet. Extensive experiments have demonstrated the promising performance of OVMR, \\eg, it outperforms existing methods across various scenarios and setups. Codes are publicly available at \\href{https://github.com/Zehong-Ma/OVMR}{https://github.com/Zehong-Ma/OVMR}.","url_abs":"https://arxiv.org/abs/2406.04675v1","url_pdf":"https://arxiv.org/pdf/2406.04675v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"ovmr-open-vocabulary-recognition-with-multi","repo_url":"https://github.com/zehong-ma/ovmr","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"open-vocabulary-object-detection","task_name":"Open Vocabulary Object Detection"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/open-vocabulary-object-detection-on-lvis-v1-0","task":"Open Vocabulary Object Detection","dataset":"LVIS v1.0","model":"OVMR","rank_in_archive_order":6,"of":28,"metrics":{"AP novel-LVIS base training":"34.4"},"uses_additional_data":true}],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2406.04675","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2406.04675"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/Zehong-Ma/OVMR","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran_draft_wrong":3,"ran":1,"unverified":3},"by_repo_kind":{"official":{"samples":7,"ran":4,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"98f385d847636a3e","entry":"basic_clean","repo":"Zehong-Ma/OVMR","repo_kind":"official","path":"clip/simple_tokenizer.py","file_url":"https://github.com/Zehong-Ma/OVMR/blob/HEAD/clip/simple_tokenizer.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"98f385d847636a3e"}},{"code_sha256_prefix":"ba26afd892405335","entry":"compute_ci95","repo":"Zehong-Ma/OVMR","repo_kind":"official","path":"parse_test_res.py","file_url":"https://github.com/Zehong-Ma/OVMR/blob/HEAD/parse_test_res.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ba26afd892405335"}},{"code_sha256_prefix":"d919ae32e5e4e616","entry":"get_pairs","repo":"Zehong-Ma/OVMR","repo_kind":"official","path":"clip/simple_tokenizer.py","file_url":"https://github.com/Zehong-Ma/OVMR/blob/HEAD/clip/simple_tokenizer.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d919ae32e5e4e616"}},{"code_sha256_prefix":"9542161e9640b858","entry":"whitespace_clean","repo":"Zehong-Ma/OVMR","repo_kind":"official","path":"clip/simple_tokenizer.py","file_url":"https://github.com/Zehong-Ma/OVMR/blob/HEAD/clip/simple_tokenizer.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"9542161e9640b858"}},{"code_sha256_prefix":"f9958e54c6bdfd78","entry":"build_model","repo":"Zehong-Ma/OVMR","repo_kind":"official","path":"clip/model.py","file_url":"https://github.com/Zehong-Ma/OVMR/blob/HEAD/clip/model.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f9958e54c6bdfd78"}},{"code_sha256_prefix":"66974576eeebe568","entry":"build_model_fp32","repo":"Zehong-Ma/OVMR","repo_kind":"official","path":"clip/model.py","file_url":"https://github.com/Zehong-Ma/OVMR/blob/HEAD/clip/model.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"66974576eeebe568"}},{"code_sha256_prefix":"605c4103ee2cd07b","entry":"load","repo":"Zehong-Ma/OVMR","repo_kind":"official","path":"clip/clip.py","file_url":"https://github.com/Zehong-Ma/OVMR/blob/HEAD/clip/clip.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"605c4103ee2cd07b"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}