{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/mmcode-evaluating-multi-modal-code-large","title":"MMCode: Benchmarking Multimodal Large Language Models for Code Generation with Visually Rich Programming Problems","arxiv_id":"2404.09486","date":"2024-04-15","proceeding":null,"authors":["Kaixin Li","Yuchen Tian","Qisheng Hu","Ziyang Luo","Zhiyong Huang","Jing Ma"],"abstract":"Programming often involves converting detailed and complex specifications into code, a process during which developers typically utilize visual aids to more effectively convey concepts. While recent developments in Large Multimodal Models have demonstrated remarkable abilities in visual reasoning and mathematical tasks, there is little work on investigating whether these models can effectively interpret visual elements for code generation. To this end, we present MMCode, the first multi-modal coding dataset for evaluating algorithmic problem-solving skills in visually rich contexts. MMCode contains 3,548 questions and 6,620 images collected from real-world programming challenges harvested from 10 code competition websites, presenting significant challenges due to the extreme demand for reasoning abilities. Our experiment results show that current state-of-the-art models struggle to solve these problems. The results highlight the lack of powerful vision-code models, and we hope MMCode can serve as an inspiration for future works in this domain. The data and code are publicly available at https://github.com/likaixin2000/MMCode.","url_abs":"https://arxiv.org/abs/2404.09486v2","url_pdf":"https://arxiv.org/pdf/2404.09486v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"mmcode-evaluating-multi-modal-code-large","repo_url":"https://github.com/happylkx/mmcode","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok"}},{"paper_slug":"mmcode-evaluating-multi-modal-code-large","repo_url":"https://github.com/likaixin2000/mmcode","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"none","reach":null},{"paper_slug":"mmcode-evaluating-multi-modal-code-large","repo_url":"https://github.com/yuchen814/codehalu","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":"benchmarking","task_name":"Benchmarking"},{"task_slug":"code-generation","task_name":"Code Generation"},{"task_slug":"visual-reasoning","task_name":"Visual Reasoning"}],"methods":[],"datasets_introduced":[{"slug":"mmcode","name":"MMCode","full_name":""}],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2404.09486","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2404.09486"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/yuchen814/codehalu","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/likaixin2000/mmcode","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/happylkx/mmcode","reach":{"status":"ok"}}],"summary":{"ran":11,"ran_draft_wrong":2,"unverified":3},"by_repo_kind":{"official":{"samples":16,"ran":13,"repositories":2}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":16,"samples":[{"code_sha256_prefix":"e61e8cf20c65c38e","entry":"auto_retry","repo":"happylkx/mmcode","repo_kind":"official","path":"dev/cralwers/crawl_codeforces.py","file_url":"https://github.com/happylkx/mmcode/blob/HEAD/dev/cralwers/crawl_codeforces.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"e61e8cf20c65c38e"}},{"code_sha256_prefix":"2def3c7eca32ab32","entry":"build_problem_graphql","repo":"happylkx/mmcode","repo_kind":"official","path":"dev/cralwers/crawl_leetcode.py","file_url":"https://github.com/happylkx/mmcode/blob/HEAD/dev/cralwers/crawl_leetcode.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"2def3c7eca32ab32"}},{"code_sha256_prefix":"2f44b77cba4f7d60","entry":"convert_base64_to_pil_image","repo":"happylkx/mmcode","repo_kind":"official","path":"utils.py","file_url":"https://github.com/happylkx/mmcode/blob/HEAD/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"2f44b77cba4f7d60"}},{"code_sha256_prefix":"b6dbf724b472263e","entry":"convert_pil_image_to_base64","repo":"happylkx/mmcode","repo_kind":"official","path":"models.py","file_url":"https://github.com/happylkx/mmcode/blob/HEAD/models.py","link_basis":"plan_row","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"b6dbf724b472263e"}},{"code_sha256_prefix":"411f6099bf0f9c3b","entry":"create_module_from_string","repo":"happylkx/mmcode","repo_kind":"official","path":"testing_utils.py","file_url":"https://github.com/happylkx/mmcode/blob/HEAD/testing_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"411f6099bf0f9c3b"}},{"code_sha256_prefix":"cc054f0a84bb15b0","entry":"custom_compare_","repo":"happylkx/mmcode","repo_kind":"official","path":"testing_utils.py","file_url":"https://github.com/happylkx/mmcode/blob/HEAD/testing_utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"cc054f0a84bb15b0"}},{"code_sha256_prefix":"46b26388baf0f1bc","entry":"extract_json_item","repo":"happylkx/mmcode","repo_kind":"official","path":"dev/cralwers/crawl_geeksforgeeks_match_taco.py","file_url":"https://github.com/happylkx/mmcode/blob/HEAD/dev/cralwers/crawl_geeksforgeeks_match_taco.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"46b26388baf0f1bc"}},{"code_sha256_prefix":"b8f69193d0fe6653","entry":"format_space","repo":"happylkx/mmcode","repo_kind":"official","path":"dev/cralwers/crawl_codeforces.py","file_url":"https://github.com/happylkx/mmcode/blob/HEAD/dev/cralwers/crawl_codeforces.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"b8f69193d0fe6653"}},{"code_sha256_prefix":"060d68859a97da44","entry":"generate_prompt","repo":"likaixin2000/mmcode","repo_kind":"official","path":"generate.py","file_url":"https://github.com/likaixin2000/mmcode/blob/HEAD/generate.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"060d68859a97da44"}},{"code_sha256_prefix":"f1767270c634a258","entry":"load_problems_from_folder","repo":"happylkx/mmcode","repo_kind":"official","path":"utils.py","file_url":"https://github.com/happylkx/mmcode/blob/HEAD/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"f1767270c634a258"}},{"code_sha256_prefix":"476124a89c12a42b","entry":"load_problems_from_jsonl","repo":"happylkx/mmcode","repo_kind":"official","path":"utils.py","file_url":"https://github.com/happylkx/mmcode/blob/HEAD/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"476124a89c12a42b"}},{"code_sha256_prefix":"6b1008f2d02c5834","entry":"read_problemset_from_file","repo":"happylkx/mmcode","repo_kind":"official","path":"dev/cralwers/crawl_codechef_match_taco.py","file_url":"https://github.com/happylkx/mmcode/blob/HEAD/dev/cralwers/crawl_codechef_match_taco.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"6b1008f2d02c5834"}},{"code_sha256_prefix":"4a94e7342cb05fb9","entry":"resize","repo":"happylkx/mmcode","repo_kind":"official","path":"models.py","file_url":"https://github.com/happylkx/mmcode/blob/HEAD/models.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"4a94e7342cb05fb9"}},{"code_sha256_prefix":"6f968344aa35117c","entry":"run_test","repo":"happylkx/mmcode","repo_kind":"official","path":"testing_utils.py","file_url":"https://github.com/happylkx/mmcode/blob/HEAD/testing_utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"6f968344aa35117c"}},{"code_sha256_prefix":"d2c98394cb5d8d94","entry":"scrape_solutions","repo":"happylkx/mmcode","repo_kind":"official","path":"dev/cralwers/crawl_aizu.py","file_url":"https://github.com/happylkx/mmcode/blob/HEAD/dev/cralwers/crawl_aizu.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"d2c98394cb5d8d94"}},{"code_sha256_prefix":"4d35a23d089b7e8a","entry":"scrape_tests","repo":"happylkx/mmcode","repo_kind":"official","path":"dev/cralwers/crawl_aizu.py","file_url":"https://github.com/happylkx/mmcode/blob/HEAD/dev/cralwers/crawl_aizu.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"4d35a23d089b7e8a"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}