{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/mmlu-pro-a-more-robust-and-challenging-multi","title":"MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmark","arxiv_id":"2406.01574","date":"2024-06-03","proceeding":null,"authors":["YuBo Wang","Xueguang Ma","Ge Zhang","Yuansheng Ni","Abhranil Chandra","Shiguang Guo","Weiming Ren","Aaran Arulraj","Xuan He","Ziyan Jiang","Tianle Li","Max Ku","Kai Wang","Alex Zhuang","Rongqi Fan","Xiang Yue","Wenhu Chen"],"abstract":"In the age of large-scale language models, benchmarks like the Massive Multitask Language Understanding (MMLU) have been pivotal in pushing the boundaries of what AI can achieve in language comprehension and reasoning across diverse domains. However, as models continue to improve, their performance on these benchmarks has begun to plateau, making it increasingly difficult to discern differences in model capabilities. This paper introduces MMLU-Pro, an enhanced dataset designed to extend the mostly knowledge-driven MMLU benchmark by integrating more challenging, reasoning-focused questions and expanding the choice set from four to ten options. Additionally, MMLU-Pro eliminates the trivial and noisy questions in MMLU. Our experimental results show that MMLU-Pro not only raises the challenge, causing a significant drop in accuracy by 16% to 33% compared to MMLU but also demonstrates greater stability under varying prompts. With 24 different prompt styles tested, the sensitivity of model scores to prompt variations decreased from 4-5% in MMLU to just 2% in MMLU-Pro. Additionally, we found that models utilizing Chain of Thought (CoT) reasoning achieved better performance on MMLU-Pro compared to direct answering, which is in stark contrast to the findings on the original MMLU, indicating that MMLU-Pro includes more complex reasoning questions. Our assessments confirm that MMLU-Pro is a more discriminative benchmark to better track progress in the field.","url_abs":"https://arxiv.org/abs/2406.01574v6","url_pdf":"https://arxiv.org/pdf/2406.01574v6.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"mmlu-pro-a-more-robust-and-challenging-multi","repo_url":"https://github.com/tiger-ai-lab/mmlu-pro","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"mmlu-pro-a-more-robust-and-challenging-multi","repo_url":"https://github.com/wenlai-lavine/jola","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"mmlu","task_name":"MMLU"},{"task_slug":"multi-task-language-understanding","task_name":"Multi-task Language Understanding"}],"methods":[{"method_slug":"set","method_name":"SET"}],"datasets_introduced":[{"slug":"mmlu-pro","name":"MMLU-Pro","full_name":""}],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2406.01574","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2406.01574"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/tiger-ai-lab/mmlu-pro","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/wenlai-lavine/jola","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran_fixture":2,"ran_draft_wrong":7,"unverified":3},"by_repo_kind":{"listed":{"samples":12,"ran":9,"repositories":2}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"30d7eec482ebf6b1","entry":"repeat_kv","repo":"wenlai-lavine/jola","repo_kind":"listed","path":"jola/modeling_llama.py","file_url":"https://github.com/wenlai-lavine/jola/blob/HEAD/jola/modeling_llama.py","link_basis":"harvester_set","language":"python","status":"ran_fixture","verification_level":2,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"30d7eec482ebf6b1"}},{"code_sha256_prefix":"bac65c3dafaec040","entry":"apply_rotary_pos_emb","repo":"wenlai-lavine/jola","repo_kind":"listed","path":"jola/modeling_qwen2.py","file_url":"https://github.com/wenlai-lavine/jola/blob/HEAD/jola/modeling_qwen2.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"bac65c3dafaec040"}},{"code_sha256_prefix":"f725bc2d76076485","entry":"apply_rotary_pos_emb","repo":"wenlai-lavine/jola","repo_kind":"listed","path":"jola/modeling_llama.py","file_url":"https://github.com/wenlai-lavine/jola/blob/HEAD/jola/modeling_llama.py","link_basis":"harvester_set","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"f725bc2d76076485"}},{"code_sha256_prefix":"628ba40cfdad9e0a","entry":"extract_again","repo":"tiger-ai-lab/mmlu-pro","repo_kind":"listed","path":"compute_accuracy.py","file_url":"https://github.com/tiger-ai-lab/mmlu-pro/blob/HEAD/compute_accuracy.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"628ba40cfdad9e0a"}},{"code_sha256_prefix":"1f59ab74392c3fcf","entry":"extract_answer","repo":"tiger-ai-lab/mmlu-pro","repo_kind":"listed","path":"compute_accuracy.py","file_url":"https://github.com/tiger-ai-lab/mmlu-pro/blob/HEAD/compute_accuracy.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"1f59ab74392c3fcf"}},{"code_sha256_prefix":"bfcfbf7614a49849","entry":"extract_final","repo":"tiger-ai-lab/mmlu-pro","repo_kind":"listed","path":"compute_accuracy.py","file_url":"https://github.com/tiger-ai-lab/mmlu-pro/blob/HEAD/compute_accuracy.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"bfcfbf7614a49849"}},{"code_sha256_prefix":"1b3ff879d7f6d22b","entry":"format_example","repo":"tiger-ai-lab/mmlu-pro","repo_kind":"listed","path":"evaluate_from_api.py","file_url":"https://github.com/tiger-ai-lab/mmlu-pro/blob/HEAD/evaluate_from_api.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"1b3ff879d7f6d22b"}},{"code_sha256_prefix":"c48a904c128bab0a","entry":"preprocess","repo":"tiger-ai-lab/mmlu-pro","repo_kind":"listed","path":"evaluate_from_api.py","file_url":"https://github.com/tiger-ai-lab/mmlu-pro/blob/HEAD/evaluate_from_api.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"c48a904c128bab0a"}},{"code_sha256_prefix":"b99eea6376d1e212","entry":"rotate_half","repo":"wenlai-lavine/jola","repo_kind":"listed","path":"jola/modeling_llama.py","file_url":"https://github.com/wenlai-lavine/jola/blob/HEAD/jola/modeling_llama.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"b99eea6376d1e212"}},{"code_sha256_prefix":"eaac056d48deb699","entry":"extract_common_reason_answer","repo":"wenlai-lavine/jola","repo_kind":"listed","path":"jola/evaluate.py","file_url":"https://github.com/wenlai-lavine/jola/blob/HEAD/jola/evaluate.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"eaac056d48deb699"}},{"code_sha256_prefix":"a57a6eff6220ee2e","entry":"extract_gem_answer","repo":"wenlai-lavine/jola","repo_kind":"listed","path":"jola/evaluate.py","file_url":"https://github.com/wenlai-lavine/jola/blob/HEAD/jola/evaluate.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"a57a6eff6220ee2e"}},{"code_sha256_prefix":"af54879256b59d48","entry":"extract_mmlu_pro_answer","repo":"wenlai-lavine/jola","repo_kind":"listed","path":"jola/evaluate.py","file_url":"https://github.com/wenlai-lavine/jola/blob/HEAD/jola/evaluate.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"af54879256b59d48"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}