{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/the-devil-is-in-the-fine-grained-details","title":"The devil is in the fine-grained details: Evaluating open-vocabulary object detectors for fine-grained understanding","arxiv_id":"2311.17518","date":"2023-11-29","proceeding":"CVPR 2024 1","authors":["Lorenzo Bianchi","Fabio Carrara","Nicola Messina","Claudio Gennaro","Fabrizio Falchi"],"abstract":"Recent advancements in large vision-language models enabled visual object detection in open-vocabulary scenarios, where object classes are defined in free-text formats during inference. In this paper, we aim to probe the state-of-the-art methods for open-vocabulary object detection to determine to what extent they understand fine-grained properties of objects and their parts. To this end, we introduce an evaluation protocol based on dynamic vocabulary generation to test whether models detect, discern, and assign the correct fine-grained description to objects in the presence of hard-negative classes. We contribute with a benchmark suite of increasing difficulty and probing different properties like color, pattern, and material. We further enhance our investigation by evaluating several state-of-the-art open-vocabulary object detectors using the proposed protocol and find that most existing solutions, which shine in standard open-vocabulary benchmarks, struggle to accurately capture and distinguish finer object details. We conclude the paper by highlighting the limitations of current methodologies and exploring promising research directions to overcome the discovered drawbacks. Data and code are available at https://lorebianchi98.github.io/FG-OVD/.","url_abs":"https://arxiv.org/abs/2311.17518v2","url_pdf":"https://arxiv.org/pdf/2311.17518v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"the-devil-is-in-the-fine-grained-details","repo_url":"https://github.com/lorebianchi98/fg-ovd","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"object","task_name":"Object"},{"task_slug":"object-detection","task_name":"Object Detection"},{"task_slug":"open-vocabulary-object-detection","task_name":"Open Vocabulary Object Detection"},{"task_slug":"open-vocabulary-object-detection","task_name":"Open-vocabulary object detection"},{"task_slug":"object-detection-1","task_name":"object-detection"}],"methods":[],"datasets_introduced":[{"slug":"fg-ovd","name":"FG-OVD","full_name":"Fine-Grained Open-Vocabulary object Detection benchmarks"}],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2311.17518","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2311.17518"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/lorebianchi98/fg-ovd","reach":null}],"summary":{"ran_draft_wrong":3},"by_repo_kind":{"official":{"samples":3,"ran":3,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":3,"samples":[{"code_sha256_prefix":"46323f57e9610d8e","entry":"load_object","repo":"lorebianchi98/fg-ovd","repo_kind":"official","path":"NoctOWL/eval/fg-ovd/evaluate_map.py","file_url":"https://github.com/lorebianchi98/fg-ovd/blob/HEAD/NoctOWL/eval/fg-ovd/evaluate_map.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"46323f57e9610d8e"}},{"code_sha256_prefix":"0be4cd21574acdc0","entry":"read_json","repo":"lorebianchi98/fg-ovd","repo_kind":"official","path":"NoctOWL/eval/fg-ovd/evaluate_map.py","file_url":"https://github.com/lorebianchi98/fg-ovd/blob/HEAD/NoctOWL/eval/fg-ovd/evaluate_map.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"0be4cd21574acdc0"}},{"code_sha256_prefix":"f702268bf37b191b","entry":"transform_predslist_to_dict","repo":"lorebianchi98/fg-ovd","repo_kind":"official","path":"NoctOWL/eval/fg-ovd/evaluate_map.py","file_url":"https://github.com/lorebianchi98/fg-ovd/blob/HEAD/NoctOWL/eval/fg-ovd/evaluate_map.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"f702268bf37b191b"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}