{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/long-context-llms-struggle-with-long-in","title":"Long-context LLMs Struggle with Long In-context Learning","arxiv_id":"2404.02060","date":"2024-04-02","proceeding":null,"authors":["Tianle Li","Ge Zhang","Quy Duc Do","Xiang Yue","Wenhu Chen"],"abstract":"Large Language Models (LLMs) have made significant strides in handling long sequences. Some models like Gemini could even to be capable of dealing with millions of tokens. However, their performance evaluation has largely been confined to metrics like perplexity and synthetic tasks, which may not fully capture their true abilities in more challenging, real-world scenarios. We introduce a benchmark (LongICLBench) for long in-context learning in extreme-label classification using six datasets with 28 to 174 classes and input lengths from 2K to 50K tokens. Our benchmark requires LLMs to comprehend the entire input to recognize the massive label spaces to make correct predictions. We evaluate on 15 long-context LLMs and find that they perform well on less challenging classification tasks with smaller label space and shorter demonstrations. However, they struggle with more challenging task like Discovery with 174 labels, suggesting a gap in their ability to process long, context-rich sequences. Further analysis reveals a bias towards labels presented later in the sequence and a need for improved reasoning over multiple pieces of information. Our study reveals that long context understanding and reasoning is still a challenging task for the existing LLMs. We believe LongICLBench could serve as a more realistic evaluation for the future long-context LLMs.","url_abs":"https://arxiv.org/abs/2404.02060v3","url_pdf":"https://arxiv.org/pdf/2404.02060v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"long-context-llms-struggle-with-long-in","repo_url":"https://github.com/tiger-ai-lab/longiclbench","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"long-context-llms-struggle-with-long-in","repo_url":"https://github.com/hhhuang/cag","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"2k","task_name":"2k"},{"task_slug":"in-context-learning","task_name":"In-Context Learning"},{"task_slug":"long-context-understanding","task_name":"Long-Context Understanding"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2404.02060","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2404.02060"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/tiger-ai-lab/longiclbench","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/hhhuang/cag","reach":null}],"summary":{"ran":6,"ran_violates":1},"by_repo_kind":{"official":{"samples":7,"ran":7,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"a341d8663c967193","entry":"format_discovery_prompt","repo":"tiger-ai-lab/longiclbench","repo_kind":"official","path":"banking77_infer.py","file_url":"https://github.com/tiger-ai-lab/longiclbench/blob/HEAD/banking77_infer.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a341d8663c967193"}},{"code_sha256_prefix":"7e77e9f4b0d18e87","entry":"format_discovery_prompt","repo":"tiger-ai-lab/longiclbench","repo_kind":"official","path":"discovery_infer.py","file_url":"https://github.com/tiger-ai-lab/longiclbench/blob/HEAD/discovery_infer.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"7e77e9f4b0d18e87"}},{"code_sha256_prefix":"9f398117ae453e86","entry":"format_long_prompt","repo":"tiger-ai-lab/longiclbench","repo_kind":"official","path":"dialoguere_infer.py","file_url":"https://github.com/tiger-ai-lab/longiclbench/blob/HEAD/dialoguere_infer.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"9f398117ae453e86"}},{"code_sha256_prefix":"cf83a84f693a8da2","entry":"intersection","repo":"tiger-ai-lab/longiclbench","repo_kind":"official","path":"data_preprocess.py","file_url":"https://github.com/tiger-ai-lab/longiclbench/blob/HEAD/data_preprocess.py","link_basis":"harvester_set","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"well_formed","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"cf83a84f693a8da2"}},{"code_sha256_prefix":"f9d4bb72caad3f36","entry":"merge_data","repo":"tiger-ai-lab/longiclbench","repo_kind":"official","path":"dialoguere_infer.py","file_url":"https://github.com/tiger-ai-lab/longiclbench/blob/HEAD/dialoguere_infer.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f9d4bb72caad3f36"}},{"code_sha256_prefix":"ce0d2cbbfe713bbb","entry":"select_data","repo":"tiger-ai-lab/longiclbench","repo_kind":"official","path":"banking77_infer.py","file_url":"https://github.com/tiger-ai-lab/longiclbench/blob/HEAD/banking77_infer.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ce0d2cbbfe713bbb"}},{"code_sha256_prefix":"351fab33fab435dc","entry":"select_test","repo":"tiger-ai-lab/longiclbench","repo_kind":"official","path":"tacred_infer.py","file_url":"https://github.com/tiger-ai-lab/longiclbench/blob/HEAD/tacred_infer.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"351fab33fab435dc"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}