{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/how-benchmark-prediction-from-fewer-data","title":"How Benchmark Prediction from Fewer Data Misses the Mark","arxiv_id":"2506.07673","date":"2025-06-09","proceeding":null,"authors":["Guanhua Zhang","Florian E. Dorner","Moritz Hardt"],"abstract":"Large language model (LLM) evaluation is increasingly costly, prompting interest in methods that speed up evaluation by shrinking benchmark datasets. Benchmark prediction (also called efficient LLM evaluation) aims to select a small subset of evaluation points and predict overall benchmark performance from that subset. In this paper, we systematically assess the strengths and limitations of 11 benchmark prediction methods across 19 diverse benchmarks. First, we identify a highly competitive baseline: Take a random sample and fit a regression model on the sample to predict missing entries. Outperforming most existing methods, this baseline challenges the assumption that careful subset selection is necessary for benchmark prediction. Second, we discover that all existing methods crucially depend on model similarity. They work best when interpolating scores among similar models. The effectiveness of benchmark prediction sharply declines when new models have higher accuracy than previously seen models. In this setting of extrapolation, none of the previous methods consistently beat a simple average over random samples. To improve over the sample average, we introduce a new method inspired by augmented inverse propensity weighting. This method consistently outperforms the random sample average even for extrapolation. However, its performance still relies on model similarity and the gains are modest in general. This shows that benchmark prediction fails just when it is most needed: at the evaluation frontier, where the goal is to evaluate new models of unknown capabilities.","url_abs":"https://arxiv.org/abs/2506.07673v1","url_pdf":"https://arxiv.org/pdf/2506.07673v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"how-benchmark-prediction-from-fewer-data","repo_url":"https://github.com/socialfoundations/benchmark-prediction","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"none","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"large-language-model","task_name":"Large Language Model"},{"task_slug":"prediction","task_name":"Prediction"}],"methods":[{"method_slug":"speed","method_name":"SPEED"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2506.07673","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2506.07673"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/socialfoundations/benchmark-prediction","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":3,"unverified":5},"by_repo_kind":{"official":{"samples":8,"ran":3,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"7a76217538ac9dff","entry":"compute_fisher_information","repo":"socialfoundations/benchmark-prediction","repo_kind":"official","path":"benchpred/metabench.py","file_url":"https://github.com/socialfoundations/benchmark-prediction/blob/HEAD/benchpred/metabench.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"7a76217538ac9dff"}},{"code_sha256_prefix":"a3462b9e34f9705d","entry":"item_curve","repo":"socialfoundations/benchmark-prediction","repo_kind":"official","path":"benchpred/tiny_bench.py","file_url":"https://github.com/socialfoundations/benchmark-prediction/blob/HEAD/benchpred/tiny_bench.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a3462b9e34f9705d"}},{"code_sha256_prefix":"a01687b77f001026","entry":"sigmoid","repo":"socialfoundations/benchmark-prediction","repo_kind":"official","path":"benchpred/tiny_bench.py","file_url":"https://github.com/socialfoundations/benchmark-prediction/blob/HEAD/benchpred/tiny_bench.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a01687b77f001026"}},{"code_sha256_prefix":"8dfd16b274b0e6b8","entry":"download_json","repo":"socialfoundations/benchmark-prediction","repo_kind":"official","path":"data_utils.py","file_url":"https://github.com/socialfoundations/benchmark-prediction/blob/HEAD/data_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"8dfd16b274b0e6b8"}},{"code_sha256_prefix":"045f9a3bcc92cae9","entry":"estimate_abilities","repo":"socialfoundations/benchmark-prediction","repo_kind":"official","path":"benchpred/metabench.py","file_url":"https://github.com/socialfoundations/benchmark-prediction/blob/HEAD/benchpred/metabench.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"045f9a3bcc92cae9"}},{"code_sha256_prefix":"8609dfefb4d496bd","entry":"estimate_ability_parameters","repo":"socialfoundations/benchmark-prediction","repo_kind":"official","path":"benchpred/tiny_bench.py","file_url":"https://github.com/socialfoundations/benchmark-prediction/blob/HEAD/benchpred/tiny_bench.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"8609dfefb4d496bd"}},{"code_sha256_prefix":"e507fcbfe732e8eb","entry":"fit_irt","repo":"socialfoundations/benchmark-prediction","repo_kind":"official","path":"benchpred/metabench.py","file_url":"https://github.com/socialfoundations/benchmark-prediction/blob/HEAD/benchpred/metabench.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e507fcbfe732e8eb"}},{"code_sha256_prefix":"565dfdf9a68e9bc5","entry":"load_glue_predictions","repo":"socialfoundations/benchmark-prediction","repo_kind":"official","path":"data_utils.py","file_url":"https://github.com/socialfoundations/benchmark-prediction/blob/HEAD/data_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"565dfdf9a68e9bc5"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}