{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/big-bench-extra-hard","title":"BIG-Bench Extra Hard","arxiv_id":"2502.19187","date":"2025-02-26","proceeding":null,"authors":["Mehran Kazemi","Bahare Fatemi","Hritik Bansal","John Palowitch","Chrysovalantis Anastasiou","Sanket Vaibhav Mehta","Lalit K. Jain","Virginia Aglietti","Disha Jindal","Peter Chen","Nishanth Dikkala","Gladys Tyen","Xin Liu","Uri Shalit","Silvia Chiappa","Kate Olszewska","Yi Tay","Vinh Q. Tran","Quoc V. Le","Orhan Firat"],"abstract":"Large language models (LLMs) are increasingly deployed in everyday applications, demanding robust general reasoning capabilities and diverse reasoning skillset. However, current LLM reasoning benchmarks predominantly focus on mathematical and coding abilities, leaving a gap in evaluating broader reasoning proficiencies. One particular exception is the BIG-Bench dataset, which has served as a crucial benchmark for evaluating the general reasoning capabilities of LLMs, thanks to its diverse set of challenging tasks that allowed for a comprehensive assessment of general reasoning across various skills within a unified framework. However, recent advances in LLMs have led to saturation on BIG-Bench, and its harder version BIG-Bench Hard (BBH). State-of-the-art models achieve near-perfect scores on many tasks in BBH, thus diminishing its utility. To address this limitation, we introduce BIG-Bench Extra Hard (BBEH), a new benchmark designed to push the boundaries of LLM reasoning evaluation. BBEH replaces each task in BBH with a novel task that probes a similar reasoning capability but exhibits significantly increased difficulty. We evaluate various models on BBEH and observe a (harmonic) average accuracy of 9.8\\% for the best general-purpose model and 44.8\\% for the best reasoning-specialized model, indicating substantial room for improvement and highlighting the ongoing challenge of achieving robust general reasoning in LLMs. We release BBEH publicly at: https://github.com/google-deepmind/bbeh.","url_abs":"https://arxiv.org/abs/2502.19187v2","url_pdf":"https://arxiv.org/pdf/2502.19187v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"big-bench-extra-hard","repo_url":"https://github.com/google-deepmind/bbeh","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"none","reach":null}],"tasks":[],"methods":[{"method_slug":"focus","method_name":"Focus"},{"method_slug":"set","method_name":"SET"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2502.19187","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2502.19187"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/google-deepmind/bbeh","reach":null}],"summary":{"ran_draft_wrong":2,"ran_violates":1},"by_repo_kind":{"official":{"samples":3,"ran":3,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"1148b66bf0848e3d","entry":"extract_answer","repo":"google-deepmind/bbeh","repo_kind":"official","path":"bbeh/evaluate.py","file_url":"https://github.com/google-deepmind/bbeh/blob/HEAD/bbeh/evaluate.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"1148b66bf0848e3d"}},{"code_sha256_prefix":"7b7bb5938764a909","entry":"fuzzy_match","repo":"google-deepmind/bbeh","repo_kind":"official","path":"bbeh/evaluate.py","file_url":"https://github.com/google-deepmind/bbeh/blob/HEAD/bbeh/evaluate.py","link_basis":"first_harvest_node","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"7b7bb5938764a909"}},{"code_sha256_prefix":"0892f960b112ef52","entry":"strip_latex","repo":"google-deepmind/bbeh","repo_kind":"official","path":"bbeh/evaluate.py","file_url":"https://github.com/google-deepmind/bbeh/blob/HEAD/bbeh/evaluate.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"0892f960b112ef52"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}