{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/mmsci-a-multimodal-multi-discipline-dataset","title":"MMSci: A Dataset for Graduate-Level Multi-Discipline Multimodal Scientific Understanding","arxiv_id":"2407.04903","date":"2024-07-06","proceeding":null,"authors":["Zekun Li","Xianjun Yang","Kyuri Choi","Wanrong Zhu","Ryan Hsieh","HyeonJung Kim","Jin Hyuk Lim","Sungyoung Ji","Byungju Lee","Xifeng Yan","Linda Ruth Petzold","Stephen D. Wilson","Woosang Lim","William Yang Wang"],"abstract":"The rapid development of Multimodal Large Language Models (MLLMs) is making AI-driven scientific assistants increasingly feasible, with interpreting scientific figures being a crucial task. However, existing datasets and benchmarks focus mainly on basic charts and limited science subjects, lacking comprehensive evaluations. To address this, we curated a multimodal, multidisciplinary dataset from peer-reviewed, open-access Nature Communications articles, spanning 72 scientific disciplines. This dataset includes figures such as schematic diagrams, simulated images, macroscopic/microscopic photos, and experimental visualizations (e.g., western blots), which often require graduate-level, discipline-specific expertise to interpret. We developed benchmarks for scientific figure captioning and multiple-choice questions, evaluating six proprietary and over ten open-source models across varied settings. The results highlight the high difficulty of these tasks and the significant performance gap among models. While many open-source models performed at chance level on the multiple-choice task, some matched the performance of proprietary models. However, the gap was more pronounced in the captioning task. Our dataset also provide valuable resource for training. Fine-tuning the Qwen2-VL-2B model with our task-specific multimodal training data improved its multiple-choice accuracy to a level comparable to GPT-4o, though captioning remains challenging. Continuous pre-training of MLLMs using our interleaved article and figure data enhanced their material generation capabilities, demonstrating potential for integrating scientific knowledge. The dataset and benchmarks will be released to support further research.","url_abs":"https://arxiv.org/abs/2407.04903v2","url_pdf":"https://arxiv.org/pdf/2407.04903v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"mmsci-a-multimodal-multi-discipline-dataset","repo_url":"https://github.com/leezekun/mmsci","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":"articles","task_name":"Articles"},{"task_slug":"instruction-following","task_name":"Instruction Following"},{"task_slug":"multiple-choice","task_name":"Multiple-choice"},{"task_slug":"visual-instruction-following","task_name":"visual instruction following"}],"methods":[{"method_slug":"focus","method_name":"Focus"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2407.04903","atlas_url":"https://app.syntology.ai/?focus=2407.04903","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2407.04903"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/leezekun/mmsci","reach":{"status":"ok"}}],"summary":{"ran":7,"ran_draft_wrong":1,"ran_fixture":1,"ran_violates":1,"ran_honours":1,"unverified":5},"by_repo_kind":{"official":{"samples":16,"ran":11,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":16,"samples":[{"code_sha256_prefix":"57e95166ade5f25c","entry":"build_transform","repo":"leezekun/mmsci","repo_kind":"official","path":"mmsci-exps/model_loader.py","file_url":"https://github.com/leezekun/mmsci/blob/HEAD/mmsci-exps/model_loader.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"57e95166ade5f25c"}},{"code_sha256_prefix":"f41cb1a19b154297","entry":"encode_image","repo":"leezekun/mmsci","repo_kind":"official","path":"mmsci-exps/model_loader.py","file_url":"https://github.com/leezekun/mmsci/blob/HEAD/mmsci-exps/model_loader.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"f41cb1a19b154297"}},{"code_sha256_prefix":"d14ec1d960b2878e","entry":"extract_all_images","repo":"leezekun/mmsci","repo_kind":"official","path":"mmsci-exps/eval/clipscore.py","file_url":"https://github.com/leezekun/mmsci/blob/HEAD/mmsci-exps/eval/clipscore.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"d14ec1d960b2878e"}},{"code_sha256_prefix":"71bae009f83e8ab5","entry":"extract_float","repo":"leezekun/mmsci","repo_kind":"official","path":"mmsci-exps/eval/factscore.py","file_url":"https://github.com/leezekun/mmsci/blob/HEAD/mmsci-exps/eval/factscore.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"71bae009f83e8ab5"}},{"code_sha256_prefix":"89e124bad3578261","entry":"extract_int","repo":"leezekun/mmsci","repo_kind":"official","path":"mmsci-exps/eval/factscore.py","file_url":"https://github.com/leezekun/mmsci/blob/HEAD/mmsci-exps/eval/factscore.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"89e124bad3578261"}},{"code_sha256_prefix":"bd77f5f8067f18e9","entry":"find_closest_aspect_ratio","repo":"leezekun/mmsci","repo_kind":"official","path":"mmsci-exps/model_loader.py","file_url":"https://github.com/leezekun/mmsci/blob/HEAD/mmsci-exps/model_loader.py","link_basis":"harvester_set","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"bd77f5f8067f18e9"}},{"code_sha256_prefix":"ea163861d7ee78cb","entry":"gemini_chat_completion","repo":"leezekun/mmsci","repo_kind":"official","path":"mmsci-exps/llm_utils.py","file_url":"https://github.com/leezekun/mmsci/blob/HEAD/mmsci-exps/llm_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"ea163861d7ee78cb"}},{"code_sha256_prefix":"195263fbff110413","entry":"llm_openai_judge","repo":"leezekun/mmsci","repo_kind":"official","path":"mmsci-exps/eval/llm_judge_scores.py","file_url":"https://github.com/leezekun/mmsci/blob/HEAD/mmsci-exps/eval/llm_judge_scores.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"195263fbff110413"}},{"code_sha256_prefix":"b9570a1d88726741","entry":"renormalize_score","repo":"leezekun/mmsci","repo_kind":"official","path":"mmsci-exps/eval/l3score.py","file_url":"https://github.com/leezekun/mmsci/blob/HEAD/mmsci-exps/eval/l3score.py","link_basis":"plan_row","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"well_formed","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"b9570a1d88726741"}},{"code_sha256_prefix":"f3988429599d6787","entry":"rule_based_answer","repo":"leezekun/mmsci","repo_kind":"official","path":"mmsci-exps/print_result.py","file_url":"https://github.com/leezekun/mmsci/blob/HEAD/mmsci-exps/print_result.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"f3988429599d6787"}},{"code_sha256_prefix":"cb11563cc362f92c","entry":"score_openai","repo":"leezekun/mmsci","repo_kind":"official","path":"mmsci-exps/eval/l3score.py","file_url":"https://github.com/leezekun/mmsci/blob/HEAD/mmsci-exps/eval/l3score.py","link_basis":"plan_row","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"cb11563cc362f92c"}},{"code_sha256_prefix":"20662984adf96374","entry":"generate_atomic_facts","repo":"leezekun/mmsci","repo_kind":"official","path":"mmsci-exps/eval/factscore.py","file_url":"https://github.com/leezekun/mmsci/blob/HEAD/mmsci-exps/eval/factscore.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"20662984adf96374"}},{"code_sha256_prefix":"0ec653aebd6101b3","entry":"get_all_metrics","repo":"leezekun/mmsci","repo_kind":"official","path":"mmsci-exps/eval/llm_judge_scores.py","file_url":"https://github.com/leezekun/mmsci/blob/HEAD/mmsci-exps/eval/llm_judge_scores.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"0ec653aebd6101b3"}},{"code_sha256_prefix":"f1a938fafaedf31d","entry":"iterate_json_file","repo":"leezekun/mmsci","repo_kind":"official","path":"mmsci-exps/eval/l3score.py","file_url":"https://github.com/leezekun/mmsci/blob/HEAD/mmsci-exps/eval/l3score.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"f1a938fafaedf31d"}},{"code_sha256_prefix":"2d7a75f8cfae958b","entry":"llm_gemini_judge","repo":"leezekun/mmsci","repo_kind":"official","path":"mmsci-exps/eval/llm_judge_scores.py","file_url":"https://github.com/leezekun/mmsci/blob/HEAD/mmsci-exps/eval/llm_judge_scores.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"2d7a75f8cfae958b"}},{"code_sha256_prefix":"c9a19ac61efdcbf7","entry":"openai_chat_completion","repo":"leezekun/mmsci","repo_kind":"official","path":"mmsci-exps/llm_utils.py","file_url":"https://github.com/leezekun/mmsci/blob/HEAD/mmsci-exps/llm_utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"c9a19ac61efdcbf7"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}