{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/are-we-done-with-mmlu","title":"Are We Done with MMLU?","arxiv_id":"2406.04127","date":"2024-06-06","proceeding":null,"authors":["Aryo Pradipta Gema","Joshua Ong Jun Leang","Giwon Hong","Alessio Devoto","Alberto Carlo Maria Mancino","Rohit Saxena","Xuanli He","Yu Zhao","Xiaotang Du","Mohammad Reza Ghasemi Madani","Claire Barale","Robert McHardy","Joshua Harris","Jean Kaddour","Emile van Krieken","Pasquale Minervini"],"abstract":"Maybe not. We identify and analyse errors in the popular Massive Multitask Language Understanding (MMLU) benchmark. Even though MMLU is widely adopted, our analysis demonstrates numerous ground truth errors that obscure the true capabilities of LLMs. For example, we find that 57% of the analysed questions in the Virology subset contain errors. To address this issue, we introduce a comprehensive framework for identifying dataset errors using a novel error annotation protocol. Then, we create MMLU-Redux, which is a subset of 5,700 manually re-annotated questions across all 57 MMLU subjects. We estimate that 6.49% of MMLU questions contain errors. Using MMLU-Redux, we demonstrate significant discrepancies with the model performance metrics that were originally reported. Our results strongly advocate for revising MMLU's error-ridden questions to enhance its future utility and reliability as a benchmark. https://huggingface.co/datasets/edinburgh-dawg/mmlu-redux-2.0.","url_abs":"https://arxiv.org/abs/2406.04127v3","url_pdf":"https://arxiv.org/pdf/2406.04127v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"are-we-done-with-mmlu","repo_url":"https://github.com/aryopg/mmlu-redux","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"are-we-done-with-mmlu","repo_url":"https://github.com/wildeval/zeroeval","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok","spdx":"Apache-2.0"}},{"paper_slug":"are-we-done-with-mmlu","repo_url":"https://github.com/yuchenlin/zeroeval","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"mmlu","task_name":"MMLU"},{"task_slug":"virology","task_name":"Virology"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2406.04127","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2406.04127"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/wildeval/zeroeval","reach":{"status":"ok","spdx":"Apache-2.0"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/yuchenlin/zeroeval","reach":{"status":"ok","spdx":"Apache-2.0"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/aryopg/mmlu-redux","reach":null}],"summary":{"ran_draft_wrong":1,"ran":1,"unverified":6},"by_repo_kind":{"listed":{"samples":7,"ran":1,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":1,"samples":[{"code_sha256_prefix":"7d2a2984ad40114e","entry":"create_model_path","repo":null,"repo_kind":null,"path":null,"file_url":null,"link_basis":"identical_code_first_harvested_elsewhere","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":null,"inline_ok":false,"mcp_get_code":{"code_sha256":"7d2a2984ad40114e"}},{"code_sha256_prefix":"8526e7d6eb6a6bd3","entry":"sanitize_args","repo":"wildeval/zeroeval","repo_kind":"listed","path":"src/unified_infer.py","file_url":"https://github.com/wildeval/zeroeval/blob/HEAD/src/unified_infer.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"8526e7d6eb6a6bd3"}},{"code_sha256_prefix":"27c94753060ee864","entry":"clear_output","repo":"wildeval/zeroeval","repo_kind":"listed","path":"src/unified_utils.py","file_url":"https://github.com/wildeval/zeroeval/blob/HEAD/src/unified_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"27c94753060ee864"}},{"code_sha256_prefix":"ff5fe0e6642b312e","entry":"generate_choice_string","repo":"wildeval/zeroeval","repo_kind":"listed","path":"src/_TEMPLATES.py","file_url":"https://github.com/wildeval/zeroeval/blob/HEAD/src/_TEMPLATES.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"ff5fe0e6642b312e"}},{"code_sha256_prefix":"2e17447aecaa3b2e","entry":"get_conv_template","repo":"wildeval/zeroeval","repo_kind":"listed","path":"src/fastchat_conversation.py","file_url":"https://github.com/wildeval/zeroeval/blob/HEAD/src/fastchat_conversation.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"2e17447aecaa3b2e"}},{"code_sha256_prefix":"041b90250a61bffa","entry":"infer_maybe_lora","repo":"wildeval/zeroeval","repo_kind":"listed","path":"src/unified_infer.py","file_url":"https://github.com/wildeval/zeroeval/blob/HEAD/src/unified_infer.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"041b90250a61bffa"}},{"code_sha256_prefix":"cfd68db505da9692","entry":"mapping_task_names","repo":"wildeval/zeroeval","repo_kind":"listed","path":"src/task_configs.py","file_url":"https://github.com/wildeval/zeroeval/blob/HEAD/src/task_configs.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"cfd68db505da9692"}},{"code_sha256_prefix":"e7a65f4765f03bd1","entry":"result_format","repo":"wildeval/zeroeval","repo_kind":"listed","path":"src/task_configs.py","file_url":"https://github.com/wildeval/zeroeval/blob/HEAD/src/task_configs.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"e7a65f4765f03bd1"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}