{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/when-benchmarks-are-targets-revealing-the","title":"When Benchmarks are Targets: Revealing the Sensitivity of Large Language Model Leaderboards","arxiv_id":"2402.01781","date":"2024-02-01","proceeding":null,"authors":["Norah Alzahrani","Hisham Abdullah Alyahya","Yazeed Alnumay","Sultan Alrashed","Shaykhah Alsubaie","Yusef Almushaykeh","Faisal Mirza","Nouf Alotaibi","Nora AlTwairesh","Areeb Alowisheq","M Saiful Bari","Haidar Khan"],"abstract":"Large Language Model (LLM) leaderboards based on benchmark rankings are regularly used to guide practitioners in model selection. Often, the published leaderboard rankings are taken at face value - we show this is a (potentially costly) mistake. Under existing leaderboards, the relative performance of LLMs is highly sensitive to (often minute) details. We show that for popular multiple-choice question benchmarks (e.g., MMLU), minor perturbations to the benchmark, such as changing the order of choices or the method of answer selection, result in changes in rankings up to 8 positions. We explain this phenomenon by conducting systematic experiments over three broad categories of benchmark perturbations and identifying the sources of this behavior. Our analysis results in several best-practice recommendations, including the advantage of a hybrid scoring method for answer selection. Our study highlights the dangers of relying on simple benchmark evaluations and charts the path for more robust evaluation schemes on the existing benchmarks. The code for this paper is available at https://github.com/National-Center-for-AI-Saudi-Arabia/lm-evaluation-harness.","url_abs":"https://arxiv.org/abs/2402.01781v2","url_pdf":"https://arxiv.org/pdf/2402.01781v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"when-benchmarks-are-targets-revealing-the","repo_url":"https://github.com/national-center-for-ai-saudi-arabia/lm-evaluation-harness","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"jax","reach":null}],"tasks":[{"task_slug":"answer-selection","task_name":"Answer Selection"},{"task_slug":"language-modeling","task_name":"Language Modeling"},{"task_slug":"language-modelling","task_name":"Language Modelling"},{"task_slug":"large-language-model","task_name":"Large Language Model"},{"task_slug":"mmlu","task_name":"MMLU"},{"task_slug":"model-selection","task_name":"Model Selection"},{"task_slug":"multiple-choice","task_name":"Multiple-choice"},{"task_slug":"sensitivity","task_name":"Sensitivity"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2402.01781","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2402.01781"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/national-center-for-ai-saudi-arabia/lm-evaluation-harness","reach":null}],"summary":{"ran_honours":1,"ran_draft_wrong":1},"by_repo_kind":{"official":{"samples":2,"ran":2,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"25c98717dbac36c9","entry":"calculate_z_value","repo":"national-center-for-ai-saudi-arabia/lm-evaluation-harness","repo_kind":"official","path":"scripts/model_comparator.py","file_url":"https://github.com/national-center-for-ai-saudi-arabia/lm-evaluation-harness/blob/HEAD/scripts/model_comparator.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"25c98717dbac36c9"}},{"code_sha256_prefix":"4cf4873400b2f793","entry":"print_results","repo":"national-center-for-ai-saudi-arabia/lm-evaluation-harness","repo_kind":"official","path":"scripts/model_comparator.py","file_url":"https://github.com/national-center-for-ai-saudi-arabia/lm-evaluation-harness/blob/HEAD/scripts/model_comparator.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"4cf4873400b2f793"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}