{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/stress-testing-long-context-language-models","title":"Stress-Testing Long-Context Language Models with Lifelong ICL and Task Haystack","arxiv_id":"2407.16695","date":"2024-07-23","proceeding":null,"authors":["Xiaoyue Xu","Qinyuan Ye","Xiang Ren"],"abstract":"We introduce Lifelong ICL, a problem setting that challenges long-context language models (LMs) to learn a sequence of language tasks through in-context learning (ICL). We further introduce Task Haystack, an evaluation suite dedicated to assessing and diagnosing how long-context LMs utilizes contexts in Lifelong ICL. When given a task instruction and test inputs, long-context LMs are expected to leverage the relevant demonstrations in the Lifelong ICL prompt, avoid distraction and interference from other tasks, and achieve test accuracies that are not significantly worse than those of the Single-task ICL baseline. Task Haystack draws inspiration from the widely-adopted \"needle-in-a-haystack\" (NIAH) evaluation, but presents distinct new challenges. It requires models (1) to utilize the contexts at a deeper level, rather than resorting to simple copying and pasting; (2) to navigate through long streams of evolving topics and tasks, proxying the complexities and dynamism of contexts in real-world scenarios. Additionally, Task Haystack inherits the controllability of NIAH, providing model developers with tools and visualizations to identify model vulnerabilities effectively. We benchmark 14 long-context LMs using Task Haystack, finding that frontier models like GPT-4o still struggle with the setting, failing on 15% of cases on average. Most open-weight models further lack behind by a large margin, with failure rates reaching up to 61%. In our controlled analysis, we identify factors such as distraction and recency bias as contributors to these failure cases. Further, performance declines when task instructions are paraphrased at test time or when ICL demonstrations are repeated excessively, raising concerns about the robustness, instruction understanding, and true context utilization of long-context LMs.","url_abs":"https://arxiv.org/abs/2407.16695v2","url_pdf":"https://arxiv.org/pdf/2407.16695v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"stress-testing-long-context-language-models","repo_url":"https://github.com/ink-usc/lifelong-icl","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"none","reach":null}],"tasks":[{"task_slug":"in-context-learning","task_name":"In-Context Learning"},{"task_slug":"navigate","task_name":"Navigate"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":null,"atlas_url":"https://app.syntology.ai/?focus=2407.16695","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2407.16695"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/ink-usc/lifelong-icl","reach":null}],"summary":{"ran":3,"ran_draft_wrong":3},"by_repo_kind":{"official":{"samples":6,"ran":6,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"76c1acb67d1fed8f","entry":"Dataloader","repo":"ink-usc/lifelong-icl","repo_kind":"official","path":"dataloader/incremental_task.py","file_url":"https://github.com/ink-usc/lifelong-icl/blob/HEAD/dataloader/incremental_task.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"76c1acb67d1fed8f"}},{"code_sha256_prefix":"e68aa86c3c1abab6","entry":"IncrementalDataloader","repo":"ink-usc/lifelong-icl","repo_kind":"official","path":"dataloader/incremental_task.py","file_url":"https://github.com/ink-usc/lifelong-icl/blob/HEAD/dataloader/incremental_task.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e68aa86c3c1abab6"}},{"code_sha256_prefix":"ddc10b8f2f0154ff","entry":"SingleTaskDataloader","repo":"ink-usc/lifelong-icl","repo_kind":"official","path":"dataloader/incremental_task.py","file_url":"https://github.com/ink-usc/lifelong-icl/blob/HEAD/dataloader/incremental_task.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ddc10b8f2f0154ff"}},{"code_sha256_prefix":"79fcb5916af8f843","entry":"generate_random_permutations","repo":"ink-usc/lifelong-icl","repo_kind":"official","path":"dataloader/incremental_task.py","file_url":"https://github.com/ink-usc/lifelong-icl/blob/HEAD/dataloader/incremental_task.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"79fcb5916af8f843"}},{"code_sha256_prefix":"0ae221496d4ff083","entry":"read_json","repo":"ink-usc/lifelong-icl","repo_kind":"official","path":"dataloader/incremental_task.py","file_url":"https://github.com/ink-usc/lifelong-icl/blob/HEAD/dataloader/incremental_task.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"0ae221496d4ff083"}},{"code_sha256_prefix":"9cfdd3d5a62805fa","entry":"read_jsonl","repo":"ink-usc/lifelong-icl","repo_kind":"official","path":"dataloader/incremental_task.py","file_url":"https://github.com/ink-usc/lifelong-icl/blob/HEAD/dataloader/incremental_task.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"9cfdd3d5a62805fa"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}