{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/enhancing-cognition-and-explainability-of","title":"Enhancing Cognition and Explainability of Multimodal Foundation Models with Self-Synthesized Data","arxiv_id":"2502.14044","date":"2025-02-19","proceeding":null,"authors":["Yucheng Shi","Quanzheng Li","Jin Sun","Xiang Li","Ninghao Liu"],"abstract":"Large multimodal models (LMMs) have shown impressive capabilities in a wide range of visual tasks. However, they often struggle with fine-grained visual reasoning, failing to identify domain-specific objectives and provide justifiable explanations for their predictions. To address this, we propose a novel visual rejection sampling framework to improve the cognition and explainability of LMMs using self-synthesized data. Specifically, visual fine-tuning requires images, queries, and target answers. Our approach begins by synthesizing interpretable answers that include human-verifiable visual features. These features are based on expert-defined concepts, carefully selected based on their alignment with the image content. After each round of fine-tuning, we apply a reward model-free filtering mechanism to select the highest-quality interpretable answers for the next round of tuning. This iterative process of data synthesis and fine-tuning progressively improves the model's ability to generate accurate and reasonable explanations. Experimental results demonstrate the effectiveness of our method in improving both the accuracy and explainability of specialized visual classification tasks.","url_abs":"https://arxiv.org/abs/2502.14044v1","url_pdf":"https://arxiv.org/pdf/2502.14044v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"enhancing-cognition-and-explainability-of","repo_url":"https://github.com/sycny/selfsynthx","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"fine-grained-visual-recognition","task_name":"Fine-Grained Visual Recognition"},{"task_slug":"pneumonia-detection","task_name":"Pneumonia Detection"},{"task_slug":"visual-reasoning","task_name":"Visual Reasoning"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/fine-grained-visual-recognition-on-cub-200-1","task":"Fine-Grained Visual Recognition","dataset":"CUB-200-2011","model":"Selfsynthx","rank_in_archive_order":1,"of":1,"metrics":{"Accuracy (%)":"85.02"},"uses_additional_data":false},{"leaderboard":"/sota/fine-grained-visual-recognition-on-fgvc-2","task":"Fine-Grained Visual Recognition","dataset":"FGVC-Aircraft","model":"Selfsynthx","rank_in_archive_order":1,"of":1,"metrics":{"Accuracy (%)":"91.99"},"uses_additional_data":false},{"leaderboard":"/sota/fine-grained-visual-recognition-on-new-plant","task":"Fine-Grained Visual Recognition","dataset":"New Plant Diseases Dataset","model":"Selfsynthx","rank_in_archive_order":1,"of":1,"metrics":{"Accuracy (% )":"97.16"},"uses_additional_data":false},{"leaderboard":"/sota/fine-grained-visual-recognition-on-stanford-2","task":"Fine-Grained Visual Recognition","dataset":"Stanford Dogs","model":"Selfsynthx","rank_in_archive_order":1,"of":1,"metrics":{"Accuracy (%)":"86.91"},"uses_additional_data":false},{"leaderboard":"/sota/pneumonia-detection-on-chest-x-ray-images-1","task":"Pneumonia Detection","dataset":"Chest X-ray images","model":"Selfsynthx","rank_in_archive_order":1,"of":4,"metrics":{"Accuracy":"98.72"},"uses_additional_data":false}],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2502.14044","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2502.14044"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/sycny/selfsynthx","reach":null}],"summary":{"ran_honours":1,"ran_draft_wrong":1},"by_repo_kind":{"official":{"samples":1,"ran":1,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":1,"samples":[{"code_sha256_prefix":"68efdb6e236ac7ee","entry":"compute_info_nce_scores","repo":"sycny/selfsynthx","repo_kind":"official","path":"src/step2.6_ans_select.py","file_url":"https://github.com/sycny/selfsynthx/blob/HEAD/src/step2.6_ans_select.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"68efdb6e236ac7ee"}},{"code_sha256_prefix":"20a7cc804eb22661","entry":"hash_args","repo":null,"repo_kind":null,"path":null,"file_url":null,"link_basis":"identical_code_first_harvested_elsewhere","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":null,"inline_ok":false,"mcp_get_code":{"code_sha256":"20a7cc804eb22661"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}