{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/arxiv-2608-03437","title":"Dynamically Allocating Evaluation Effort for Model Ranking","arxiv_id":"2608.03437","date":"2026-08-04","proceeding":null,"authors":["Vilém Zouhar","Julia Kreutzer","Alon Lavie","Tom Kocmi","Matt Post","Ondřej Bojar","Mrinmaya Sachan"],"abstract":"While human evaluation is the gold standard in many NLP tasks, it suffers from prohibitive costs and poor scalability. When identifying top-performing models, typical evaluation protocols waste effort by exhaustively evaluating all models on the entire benchmark, a safe but inefficient approach. In this work, we formalize multi-model human evaluation as a best-arm identification problem in a multi-armed bandit setup with correlated arms, where pulling an arm corresponds to human-evaluating a model. By sampling adaptively based on the intermediate model rankings obtained on the samples so far, we can focus the annotation budget on the most competitive models. We prove the optimality of the proposed algorithms and show that it improves discrimination between top-performing models. This makes evaluations faster, cheaper and more aligned with large-scale competition evaluation goals.","url_abs":"https://arxiv.org/abs/2608.03437","url_pdf":"https://arxiv.org/pdf/2608.03437","source":{"archive":null,"snapshot":"2025-07-28","note":"not in the Papers with Code archive (frozen at the snapshot)","row_kind":"graph","title_abstract_authors_date":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)"},"code_links":[],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":null,"mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2608.03437"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"mentioned_in_github":null,"is_official":null,"provenance":"deterministic:regex_extraction","mentioned_in_paper":null,"url":"https://github.com/zouharvi/evaluation-bandit","reach":null}],"summary":{"ran":5},"by_repo_kind":{"found_in_text":{"samples":5,"ran":5,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"55519304ca637b6f","entry":"additive","repo":"zouharvi/evaluation-bandit","repo_kind":"found_in_text","path":"evaluation_bandit/estimators.py","file_url":"https://github.com/zouharvi/evaluation-bandit/blob/HEAD/evaluation_bandit/estimators.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"55519304ca637b6f"}},{"code_sha256_prefix":"7c9ef1cc29e8f521","entry":"pval","repo":"zouharvi/evaluation-bandit","repo_kind":"found_in_text","path":"evaluation_bandit/utils.py","file_url":"https://github.com/zouharvi/evaluation-bandit/blob/HEAD/evaluation_bandit/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"7c9ef1cc29e8f521"}},{"code_sha256_prefix":"d8378f6bf1d52cc7","entry":"pval_ind","repo":"zouharvi/evaluation-bandit","repo_kind":"found_in_text","path":"evaluation_bandit/utils.py","file_url":"https://github.com/zouharvi/evaluation-bandit/blob/HEAD/evaluation_bandit/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"d8378f6bf1d52cc7"}},{"code_sha256_prefix":"b47d5686405ca7f3","entry":"subset2evaluate_to_sorter","repo":"zouharvi/evaluation-bandit","repo_kind":"found_in_text","path":"evaluation_bandit/simulation.py","file_url":"https://github.com/zouharvi/evaluation-bandit/blob/HEAD/evaluation_bandit/simulation.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"b47d5686405ca7f3"}},{"code_sha256_prefix":"3a4d8ea5a02aa28b","entry":"tau","repo":"zouharvi/evaluation-bandit","repo_kind":"found_in_text","path":"evaluation_bandit/utils.py","file_url":"https://github.com/zouharvi/evaluation-bandit/blob/HEAD/evaluation_bandit/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"3a4d8ea5a02aa28b"}}]},"arxiv_metadata":{"licence":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)","fields":["title","abstract","authors","date"],"primary_category":"cs.LG","source":"arxiv_2026.jsonl"},"syntology_extracted_results":null}