{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/analyzing-and-boosting-the-power-of-fine","title":"Analyzing and Boosting the Power of Fine-Grained Visual Recognition for Multi-modal Large Language Models","arxiv_id":"2501.15140","date":"2025-01-25","proceeding":null,"authors":["Hulingxiao He","Geng Li","Zijun Geng","Jinglin Xu","Yuxin Peng"],"abstract":"Multi-modal large language models (MLLMs) have shown remarkable abilities in various visual understanding tasks. However, MLLMs still struggle with fine-grained visual recognition (FGVR), which aims to identify subordinate-level categories from images. This can negatively impact more advanced capabilities of MLLMs, such as object-centric visual question answering and reasoning. In our study, we revisit three quintessential capabilities of MLLMs for FGVR, including object information extraction, category knowledge reserve, object-category alignment, and position of the root cause as a misalignment problem. To address this issue, we present Finedefics, an MLLM that enhances the model's FGVR capability by incorporating informative attribute descriptions of objects into the training phase. We employ contrastive learning on object-attribute pairs and attribute-category pairs simultaneously and use examples from similar but incorrect categories as hard negatives, naturally bringing representations of visual objects and category names closer. Extensive evaluations across multiple popular FGVR datasets demonstrate that Finedefics outperforms existing MLLMs of comparable parameter sizes, showcasing its remarkable efficacy. The code is available at https://github.com/PKU-ICST-MIPL/Finedefics_ICLR2025.","url_abs":"https://arxiv.org/abs/2501.15140v3","url_pdf":"https://arxiv.org/pdf/2501.15140v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"analyzing-and-boosting-the-power-of-fine","repo_url":"https://github.com/pku-icst-mipl/finedefics_iclr2025","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"attribute","task_name":"Attribute"},{"task_slug":"contrastive-learning","task_name":"Contrastive Learning"},{"task_slug":"fine-grained-visual-recognition","task_name":"Fine-Grained Visual Recognition"},{"task_slug":"object","task_name":"Object"},{"task_slug":"question-answering","task_name":"Question Answering"},{"task_slug":"visual-question-answering-1","task_name":"Visual Question Answering"}],"methods":[{"method_slug":"contrastive-learning","method_name":"Contrastive Learning"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2501.15140","atlas_url":"https://app.syntology.ai/?focus=2501.15140","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2501.15140"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/PKU-ICST-MIPL/Finedefics","reach":{"status":"gone","observed_at":"2026-09-17","how":"tree_404+repo_404"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/pku-icst-mipl/finedefics_iclr2025","reach":null}],"summary":{"ran_draft_wrong":4,"unverified":2},"by_repo_kind":{"official":{"samples":6,"ran":4,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":6,"samples":[{"code_sha256_prefix":"0518572f1f4fc34c","entry":"model_template","repo":"pku-icst-mipl/finedefics_iclr2025","repo_kind":"official","path":"FOCI-Benchmark/benchmark/model/model.py","file_url":"https://github.com/pku-icst-mipl/finedefics_iclr2025/blob/HEAD/FOCI-Benchmark/benchmark/model/model.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"0518572f1f4fc34c"}},{"code_sha256_prefix":"97922863aafd3d20","entry":"prepare_prompt","repo":"pku-icst-mipl/finedefics_iclr2025","repo_kind":"official","path":"FOCI-Benchmark/benchmark/model/model.py","file_url":"https://github.com/pku-icst-mipl/finedefics_iclr2025/blob/HEAD/FOCI-Benchmark/benchmark/model/model.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"97922863aafd3d20"}},{"code_sha256_prefix":"ecd1c5198a8af802","entry":"prepare_prompt_mc","repo":"pku-icst-mipl/finedefics_iclr2025","repo_kind":"official","path":"FOCI-Benchmark/benchmark/model/model.py","file_url":"https://github.com/pku-icst-mipl/finedefics_iclr2025/blob/HEAD/FOCI-Benchmark/benchmark/model/model.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"ecd1c5198a8af802"}},{"code_sha256_prefix":"619ec23d435b2144","entry":"prepare_prompt_yn","repo":"pku-icst-mipl/finedefics_iclr2025","repo_kind":"official","path":"FOCI-Benchmark/benchmark/model/model.py","file_url":"https://github.com/pku-icst-mipl/finedefics_iclr2025/blob/HEAD/FOCI-Benchmark/benchmark/model/model.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"619ec23d435b2144"}},{"code_sha256_prefix":"e3705b7124f1c667","entry":"HFModel","repo":"pku-icst-mipl/finedefics_iclr2025","repo_kind":"official","path":"FOCI-Benchmark/benchmark/model/model.py","file_url":"https://github.com/pku-icst-mipl/finedefics_iclr2025/blob/HEAD/FOCI-Benchmark/benchmark/model/model.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"e3705b7124f1c667"}},{"code_sha256_prefix":"b2d44a2a2b109b46","entry":"Idefics2Model","repo":"pku-icst-mipl/finedefics_iclr2025","repo_kind":"official","path":"FOCI-Benchmark/benchmark/model/model.py","file_url":"https://github.com/pku-icst-mipl/finedefics_iclr2025/blob/HEAD/FOCI-Benchmark/benchmark/model/model.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"b2d44a2a2b109b46"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}