{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/how-to-evaluate-the-generalization-of","title":"How to Evaluate the Generalization of Detection? A Benchmark for Comprehensive Open-Vocabulary Detection","arxiv_id":"2308.13177","date":"2023-08-25","proceeding":null,"authors":["Yiyang Yao","Peng Liu","Tiancheng Zhao","Qianqian Zhang","Jiajia Liao","Chunxin Fang","Kyusong Lee","Qing Wang"],"abstract":"Object detection (OD) in computer vision has made significant progress in recent years, transitioning from closed-set labels to open-vocabulary detection (OVD) based on large-scale vision-language pre-training (VLP). However, current evaluation methods and datasets are limited to testing generalization over object types and referral expressions, which do not provide a systematic, fine-grained, and accurate benchmark of OVD models' abilities. In this paper, we propose a new benchmark named OVDEval, which includes 9 sub-tasks and introduces evaluations on commonsense knowledge, attribute understanding, position understanding, object relation comprehension, and more. The dataset is meticulously created to provide hard negatives that challenge models' true understanding of visual and linguistic input. Additionally, we identify a problem with the popular Average Precision (AP) metric when benchmarking models on these fine-grained label datasets and propose a new metric called Non-Maximum Suppression Average Precision (NMS-AP) to address this issue. Extensive experimental results show that existing top OVD models all fail on the new tasks except for simple object types, demonstrating the value of the proposed dataset in pinpointing the weakness of current OVD models and guiding future research. Furthermore, the proposed NMS-AP metric is verified by experiments to provide a much more truthful evaluation of OVD models, whereas traditional AP metrics yield deceptive results. Data is available at \\url{https://github.com/om-ai-lab/OVDEval}","url_abs":"https://arxiv.org/abs/2308.13177v2","url_pdf":"https://arxiv.org/pdf/2308.13177v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"how-to-evaluate-the-generalization-of","repo_url":"https://github.com/om-ai-lab/ovdeval","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}},{"paper_slug":"how-to-evaluate-the-generalization-of","repo_url":"https://github.com/om-ai-lab/OmAgent","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok","spdx":"Apache-2.0"}},{"paper_slug":"how-to-evaluate-the-generalization-of","repo_url":"https://github.com/om-ai-lab/OmDet","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}},{"paper_slug":"how-to-evaluate-the-generalization-of","repo_url":"https://github.com/om-ai-lab/ZoomEye","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"object-detection","task_name":"Object Detection"}],"methods":[],"datasets_introduced":[{"slug":"ovdeval","name":"OVDEval","full_name":""}],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2308.13177","atlas_url":"https://app.syntology.ai/?focus=2308.13177","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2308.13177"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/om-ai-lab/ZoomEye","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/om-ai-lab/ovdeval","reach":{"status":"ok","spdx":"Apache-2.0"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/om-ai-lab/OmAgent","reach":{"status":"ok","spdx":"Apache-2.0"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/om-ai-lab/OmDet","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran":7,"ran_draft_wrong":2,"ran_fixture":1},"by_repo_kind":{"official":{"samples":2,"ran":2,"repositories":1},"listed":{"samples":8,"ran":8,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"3e006b27eb5131f9","entry":"apply_deltas","repo":"om-ai-lab/OmDet","repo_kind":"listed","path":"omdet/modeling/common.py","file_url":"https://github.com/om-ai-lab/OmDet/blob/HEAD/omdet/modeling/common.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"3e006b27eb5131f9"}},{"code_sha256_prefix":"f37e6f7d03c74102","entry":"calculate_iou","repo":"om-ai-lab/ovdeval","repo_kind":"official","path":"eval_glip.py","file_url":"https://github.com/om-ai-lab/ovdeval/blob/HEAD/eval_glip.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"f37e6f7d03c74102"}},{"code_sha256_prefix":"dd1114865f06f0fd","entry":"conv3x3","repo":"om-ai-lab/OmDet","repo_kind":"listed","path":"omdet/modeling/backbone/dlafpn.py","file_url":"https://github.com/om-ai-lab/OmDet/blob/HEAD/omdet/modeling/backbone/dlafpn.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"dd1114865f06f0fd"}},{"code_sha256_prefix":"7e97d239afcab15e","entry":"get_fpn_config","repo":"om-ai-lab/OmDet","repo_kind":"listed","path":"omdet/modeling/backbone/bifpn.py","file_url":"https://github.com/om-ai-lab/OmDet/blob/HEAD/omdet/modeling/backbone/bifpn.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"7e97d239afcab15e"}},{"code_sha256_prefix":"738b4871dc26c54f","entry":"get_model_url","repo":"om-ai-lab/OmDet","repo_kind":"listed","path":"omdet/modeling/backbone/dlafpn.py","file_url":"https://github.com/om-ai-lab/OmDet/blob/HEAD/omdet/modeling/backbone/dlafpn.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"738b4871dc26c54f"}},{"code_sha256_prefix":"9c966a0badf6f168","entry":"get_output_shape","repo":"om-ai-lab/OmDet","repo_kind":"listed","path":"omdet/inference/base_engine.py","file_url":"https://github.com/om-ai-lab/OmDet/blob/HEAD/omdet/inference/base_engine.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"9c966a0badf6f168"}},{"code_sha256_prefix":"bdd48ffb812ebe9b","entry":"nms_threaded","repo":"om-ai-lab/ovdeval","repo_kind":"official","path":"eval_glip.py","file_url":"https://github.com/om-ai-lab/ovdeval/blob/HEAD/eval_glip.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"bdd48ffb812ebe9b"}},{"code_sha256_prefix":"717e4d22b6d103e6","entry":"swish","repo":"om-ai-lab/OmDet","repo_kind":"listed","path":"omdet/modeling/backbone/bifpn.py","file_url":"https://github.com/om-ai-lab/OmDet/blob/HEAD/omdet/modeling/backbone/bifpn.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"717e4d22b6d103e6"}},{"code_sha256_prefix":"f9fd6241d935f07b","entry":"window_partition","repo":"om-ai-lab/OmDet","repo_kind":"listed","path":"omdet/modeling/backbone/swint.py","file_url":"https://github.com/om-ai-lab/OmDet/blob/HEAD/omdet/modeling/backbone/swint.py","link_basis":"harvester_set","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"f9fd6241d935f07b"}},{"code_sha256_prefix":"fb32094c6dbece71","entry":"window_reverse","repo":"om-ai-lab/OmDet","repo_kind":"listed","path":"omdet/modeling/backbone/swint.py","file_url":"https://github.com/om-ai-lab/OmDet/blob/HEAD/omdet/modeling/backbone/swint.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"fb32094c6dbece71"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}