{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/babilong-testing-the-limits-of-llms-with-long","title":"BABILong: Testing the Limits of LLMs with Long Context Reasoning-in-a-Haystack","arxiv_id":"2406.10149","date":"2024-06-14","proceeding":null,"authors":["Yuri Kuratov","Aydar Bulatov","Petr Anokhin","Ivan Rodkin","Dmitry Sorokin","Artyom Sorokin","Mikhail Burtsev"],"abstract":"In recent years, the input context sizes of large language models (LLMs) have increased dramatically. However, existing evaluation methods have not kept pace, failing to comprehensively assess the efficiency of models in handling long contexts. To bridge this gap, we introduce the BABILong benchmark, designed to test language models' ability to reason across facts distributed in extremely long documents. BABILong includes a diverse set of 20 reasoning tasks, including fact chaining, simple induction, deduction, counting, and handling lists/sets. These tasks are challenging on their own, and even more demanding when the required facts are scattered across long natural text. Our evaluations show that popular LLMs effectively utilize only 10-20\\% of the context and their performance declines sharply with increased reasoning complexity. Among alternatives to in-context reasoning, Retrieval-Augmented Generation methods achieve a modest 60\\% accuracy on single-fact question answering, independent of context length. Among context extension methods, the highest performance is demonstrated by recurrent memory transformers after fine-tuning, enabling the processing of lengths up to 50 million tokens. The BABILong benchmark is extendable to any length to support the evaluation of new upcoming models with increased capabilities, and we provide splits up to 10 million token lengths.","url_abs":"https://arxiv.org/abs/2406.10149v2","url_pdf":"https://arxiv.org/pdf/2406.10149v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"babilong-testing-the-limits-of-llms-with-long","repo_url":"https://github.com/booydar/babilong","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}},{"paper_slug":"babilong-testing-the-limits-of-llms-with-long","repo_url":"https://github.com/InternLM/opencompass","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}},{"paper_slug":"babilong-testing-the-limits-of-llms-with-long","repo_url":"https://github.com/NVIDIA/Star-Attention","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}},{"paper_slug":"babilong-testing-the-limits-of-llms-with-long","repo_url":"https://github.com/open-compass/opencompass","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"question-answering","task_name":"Question Answering"},{"task_slug":"retrieval-augmented-generation","task_name":"Retrieval-augmented Generation"}],"methods":[{"method_slug":"set","method_name":"SET"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2406.10149","atlas_url":"https://app.syntology.ai/?focus=2406.10149","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2406.10149"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/booydar/babilong","reach":{"status":"ok","spdx":"Apache-2.0"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/NVIDIA/Star-Attention","reach":{"status":"ok","spdx":"Apache-2.0"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/open-compass/opencompass","reach":{"status":"ok","spdx":"Apache-2.0"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/InternLM/opencompass","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran":7,"unverified":1},"by_repo_kind":{"official":{"samples":8,"ran":7,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"13c862882ed08796","entry":"compare_answers","repo":"booydar/babilong","repo_kind":"official","path":"babilong/babilong_utils.py","file_url":"https://github.com/booydar/babilong/blob/HEAD/babilong/babilong_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"13c862882ed08796"}},{"code_sha256_prefix":"eebaceb4246b1413","entry":"compare_answers","repo":"booydar/babilong","repo_kind":"official","path":"babilong/metrics.py","file_url":"https://github.com/booydar/babilong/blob/HEAD/babilong/metrics.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"eebaceb4246b1413"}},{"code_sha256_prefix":"3a65b7efc6fb285d","entry":"get_formatted_input","repo":"booydar/babilong","repo_kind":"official","path":"babilong/prompts.py","file_url":"https://github.com/booydar/babilong/blob/HEAD/babilong/prompts.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"3a65b7efc6fb285d"}},{"code_sha256_prefix":"2f19bc766480b809","entry":"parse_run_cfg","repo":"booydar/babilong","repo_kind":"official","path":"babilong/collect_results.py","file_url":"https://github.com/booydar/babilong/blob/HEAD/babilong/collect_results.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"2f19bc766480b809"}},{"code_sha256_prefix":"afddb622e3acfafe","entry":"plot_results","repo":"booydar/babilong","repo_kind":"official","path":"babilong/collect_results.py","file_url":"https://github.com/booydar/babilong/blob/HEAD/babilong/collect_results.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"afddb622e3acfafe"}},{"code_sha256_prefix":"4b85c3e1781febe5","entry":"preprocess_output","repo":"booydar/babilong","repo_kind":"official","path":"babilong/metrics.py","file_url":"https://github.com/booydar/babilong/blob/HEAD/babilong/metrics.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"4b85c3e1781febe5"}},{"code_sha256_prefix":"2da2bf2533e266ad","entry":"sum_lengths","repo":"booydar/babilong","repo_kind":"official","path":"babilong/babilong_utils.py","file_url":"https://github.com/booydar/babilong/blob/HEAD/babilong/babilong_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"2da2bf2533e266ad"}},{"code_sha256_prefix":"4923e6cc1a07a2fd","entry":"get_dataset_df","repo":"booydar/babilong","repo_kind":"official","path":"babilong/babilong_utils.py","file_url":"https://github.com/booydar/babilong/blob/HEAD/babilong/babilong_utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"4923e6cc1a07a2fd"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}