{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/score-systematic-consistency-and-robustness","title":"SCORE: Systematic COnsistency and Robustness Evaluation for Large Language Models","arxiv_id":"2503.00137","date":"2025-02-28","proceeding":null,"authors":["Grigor Nalbandyan","Rima Shahbazyan","Evelina Bakhturina"],"abstract":"Typical evaluations of Large Language Models (LLMs) report a single metric per dataset, often representing the model's best-case performance under carefully selected settings. Unfortunately, this approach overlooks model robustness and reliability in real-world applications. For instance, simple paraphrasing of prompts on the MMLU-Pro dataset causes accuracy fluctuations of up to 10\\%, while reordering answer choices in the AGIEval dataset results in accuracy differences of up to 6.1\\%. While some studies discuss issues with LLM robustness, there is no unified or centralized framework for evaluating the robustness of language models. To address this gap and consolidate existing research on model robustness, we present SCORE ($\\mathbf{S}$ystematic $\\mathbf{CO}$nsistency and $\\mathbf{R}$obustness $\\mathbf{E}$valuation), a comprehensive framework for non-adversarial evaluation of LLMs. The SCORE framework evaluates models by repeatedly testing them on the same benchmarks in various setups to give a realistic estimate of their accuracy and consistency. We release the code publicly and start an LLM robustness leaderboard to facilitate further development and research.","url_abs":"https://arxiv.org/abs/2503.00137v1","url_pdf":"https://arxiv.org/pdf/2503.00137v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"score-systematic-consistency-and-robustness","repo_url":"https://github.com/EleutherAI/lm-evaluation-harness","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"jax","reach":null}],"tasks":[{"task_slug":"mmlu","task_name":"MMLU"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2503.00137","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2503.00137"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/EleutherAI/lm-evaluation-harness","reach":null},{"provenance":"deterministic:regex_extraction","url":"https://github.com/EleutherAI/lm-evaluationharness","reach":{"status":"gone","observed_at":"2026-09-16","how":"tree_404+repo_404"}}],"summary":{"ran_honours":1},"by_repo_kind":{"official":{"samples":1,"ran":1,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"9181c05917d9fdef","entry":"calculate_consistency_rate","repo":"EleutherAI/lm-evaluation-harness","repo_kind":"official","path":"lm_eval/tasks/score/utils.py","file_url":"https://github.com/EleutherAI/lm-evaluation-harness/blob/HEAD/lm_eval/tasks/score/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"9181c05917d9fdef"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}