{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/multilingual-needle-in-a-haystack","title":"Multilingual Needle in a Haystack: Investigating Long-Context Behavior of Multilingual Large Language Models","arxiv_id":"2408.10151","date":"2024-08-19","proceeding":null,"authors":["Amey Hengle","Prasoon Bajpai","Soham Dan","Tanmoy Chakraborty"],"abstract":"While recent large language models (LLMs) demonstrate remarkable abilities in responding to queries in diverse languages, their ability to handle long multilingual contexts is unexplored. As such, a systematic evaluation of the long-context capabilities of LLMs in multilingual settings is crucial, specifically in the context of information retrieval. To address this gap, we introduce the MultiLingual Needle-in-a-Haystack (MLNeedle) test, designed to assess a model's ability to retrieve relevant information (the needle) from a collection of multilingual distractor texts (the haystack). This test serves as an extension of the multilingual question-answering task, encompassing both monolingual and cross-lingual retrieval. We evaluate four state-of-the-art LLMs on MLNeedle. Our findings reveal that model performance can vary significantly with language and needle position. Specifically, we observe that model performance is the lowest when the needle is (i) in a language outside the English language family and (ii) located in the middle of the input context. Furthermore, although some models claim a context size of $8k$ tokens or greater, none demonstrate satisfactory cross-lingual retrieval performance as the context length increases. Our analysis provides key insights into the long-context behavior of LLMs in multilingual settings to guide future evaluation protocols. To our knowledge, this is the first study to investigate the multilingual long-context behavior of LLMs.","url_abs":"https://arxiv.org/abs/2408.10151v1","url_pdf":"https://arxiv.org/pdf/2408.10151v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"multilingual-needle-in-a-haystack","repo_url":"https://github.com/AmeyHengle/multilingual-needle-in-a-haystack","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":null,"task_name":"8k"},{"task_slug":"information-retrieval","task_name":"Information Retrieval"},{"task_slug":"question-answering","task_name":"Question Answering"},{"task_slug":"retrieval","task_name":"Retrieval"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2408.10151","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2408.10151"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/AmeyHengle/multilingual-needle-in-a-haystack","reach":{"status":"ok"}}],"summary":{"ran":6},"by_repo_kind":{"official":{"samples":6,"ran":6,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":6,"samples":[{"code_sha256_prefix":"815111ed1226ad3b","entry":"analysis2_context_lang","repo":"AmeyHengle/multilingual-needle-in-a-haystack","repo_kind":"official","path":"generate_plots.py","file_url":"https://github.com/AmeyHengle/multilingual-needle-in-a-haystack/blob/HEAD/generate_plots.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"815111ed1226ad3b"}},{"code_sha256_prefix":"a4bc13c92759dd11","entry":"analysis2_noise_type","repo":"AmeyHengle/multilingual-needle-in-a-haystack","repo_kind":"official","path":"generate_plots.py","file_url":"https://github.com/AmeyHengle/multilingual-needle-in-a-haystack/blob/HEAD/generate_plots.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"a4bc13c92759dd11"}},{"code_sha256_prefix":"b1d28b3222271e7f","entry":"analysis2_noise_type_mono_bi_multi","repo":"AmeyHengle/multilingual-needle-in-a-haystack","repo_kind":"official","path":"generate_plots.py","file_url":"https://github.com/AmeyHengle/multilingual-needle-in-a-haystack/blob/HEAD/generate_plots.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"b1d28b3222271e7f"}},{"code_sha256_prefix":"c818ddfd623588f7","entry":"format_inputs","repo":"AmeyHengle/multilingual-needle-in-a-haystack","repo_kind":"official","path":"inference_cohere_aya.py","file_url":"https://github.com/AmeyHengle/multilingual-needle-in-a-haystack/blob/HEAD/inference_cohere_aya.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"c818ddfd623588f7"}},{"code_sha256_prefix":"9f01b971f4b7d753","entry":"insert_needle","repo":"AmeyHengle/multilingual-needle-in-a-haystack","repo_kind":"official","path":"generate_prompts.py","file_url":"https://github.com/AmeyHengle/multilingual-needle-in-a-haystack/blob/HEAD/generate_prompts.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"9f01b971f4b7d753"}},{"code_sha256_prefix":"9e8794ff7a9d2b3a","entry":"postprocess_aya_output","repo":"AmeyHengle/multilingual-needle-in-a-haystack","repo_kind":"official","path":"inference_cohere_aya.py","file_url":"https://github.com/AmeyHengle/multilingual-needle-in-a-haystack/blob/HEAD/inference_cohere_aya.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"9e8794ff7a9d2b3a"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}