{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/never-lost-in-the-middle-improving-large","title":"Never Lost in the Middle: Mastering Long-Context Question Answering with Position-Agnostic Decompositional Training","arxiv_id":"2311.09198","date":"2023-11-15","proceeding":null,"authors":["Junqing He","Kunhao Pan","Xiaoqun Dong","Zhuoyang Song","Yibo Liu","Qianguo Sun","Yuxin Liang","Hao Wang","Enming Zhang","Jiaxing Zhang"],"abstract":"While large language models (LLMs) are equipped with longer text input capabilities than before, they are struggling to seek correct information in long contexts. The \"lost in the middle\" problem challenges most LLMs, referring to the dramatic decline in accuracy when correct information is located in the middle. To overcome this crucial issue, this paper proposes to enhance the information searching and reflection ability of LLMs in long contexts via specially designed tasks called Attention Strengthening Multi-doc QA (ASM QA). Following these tasks, our model excels in focusing more precisely on the desired information. Experimental results show substantial improvement in Multi-doc QA and other benchmarks, superior to state-of-the-art models by 13.7% absolute gain in shuffled settings, by 21.5% in passage retrieval task. We release our model, Ziya-Reader to promote related research in the community.","url_abs":"https://arxiv.org/abs/2311.09198v2","url_pdf":"https://arxiv.org/pdf/2311.09198v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"never-lost-in-the-middle-improving-large","repo_url":"https://github.com/hejunqing/never-lost-in-the-middle","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"never-lost-in-the-middle-improving-large","repo_url":"https://github.com/qihoo360/360zhinao","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"passage-retrieval","task_name":"Passage Retrieval"},{"task_slug":null,"task_name":"Position"},{"task_slug":"question-answering","task_name":"Question Answering"},{"task_slug":"retrieval","task_name":"Retrieval"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2311.09198","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2311.09198"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/qihoo360/360zhinao","reach":{"status":"ok","spdx":"Apache-2.0"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/hejunqing/never-lost-in-the-middle","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":10,"ran_honours":1,"ran_draft_wrong":3,"ran_violates":1,"unverified":2},"by_repo_kind":{"official":{"samples":15,"ran":14,"repositories":1},"listed":{"samples":2,"ran":1,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"afdae69731929a72","entry":"add_eod","repo":"hejunqing/never-lost-in-the-middle","repo_kind":"official","path":"LongBench/src/new_utils.py","file_url":"https://github.com/hejunqing/never-lost-in-the-middle/blob/HEAD/LongBench/src/new_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"afdae69731929a72"}},{"code_sha256_prefix":"251720ea85d2ca71","entry":"build_chat","repo":"hejunqing/never-lost-in-the-middle","repo_kind":"official","path":"Lost_Retrieval/src/pred_single.py","file_url":"https://github.com/hejunqing/never-lost-in-the-middle/blob/HEAD/Lost_Retrieval/src/pred_single.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"251720ea85d2ca71"}},{"code_sha256_prefix":"7024e12377186e51","entry":"build_chat","repo":"hejunqing/never-lost-in-the-middle","repo_kind":"official","path":"LongBench/src/baichuan_pred_shuffle.py","file_url":"https://github.com/hejunqing/never-lost-in-the-middle/blob/HEAD/LongBench/src/baichuan_pred_shuffle.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"7024e12377186e51"}},{"code_sha256_prefix":"7439724189daf2cb","entry":"build_chat","repo":"hejunqing/never-lost-in-the-middle","repo_kind":"official","path":"LongBench/src/pred_chatglm.py","file_url":"https://github.com/hejunqing/never-lost-in-the-middle/blob/HEAD/LongBench/src/pred_chatglm.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"7439724189daf2cb"}},{"code_sha256_prefix":"619fcfb88a0948e5","entry":"build_chat2","repo":"hejunqing/never-lost-in-the-middle","repo_kind":"official","path":"Lost_Retrieval/src/pred_single.py","file_url":"https://github.com/hejunqing/never-lost-in-the-middle/blob/HEAD/Lost_Retrieval/src/pred_single.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"619fcfb88a0948e5"}},{"code_sha256_prefix":"9cea8a9dc64a79cc","entry":"build_chat_baichuan","repo":"hejunqing/never-lost-in-the-middle","repo_kind":"official","path":"LongBench/src/pred_shuffle_reader.py","file_url":"https://github.com/hejunqing/never-lost-in-the-middle/blob/HEAD/LongBench/src/pred_shuffle_reader.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"9cea8a9dc64a79cc"}},{"code_sha256_prefix":"b349b79d9cc2934b","entry":"count_score","repo":"hejunqing/never-lost-in-the-middle","repo_kind":"official","path":"LongBench/src/metrics.py","file_url":"https://github.com/hejunqing/never-lost-in-the-middle/blob/HEAD/LongBench/src/metrics.py","link_basis":"harvester_set","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b349b79d9cc2934b"}},{"code_sha256_prefix":"9ac0485160d687f7","entry":"get_pred","repo":"hejunqing/never-lost-in-the-middle","repo_kind":"official","path":"LongBench/src/baichuan_pred_shuffle.py","file_url":"https://github.com/hejunqing/never-lost-in-the-middle/blob/HEAD/LongBench/src/baichuan_pred_shuffle.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"9ac0485160d687f7"}},{"code_sha256_prefix":"e7e75981cb464788","entry":"normalize_answer","repo":"hejunqing/never-lost-in-the-middle","repo_kind":"official","path":"LongBench/src/metrics.py","file_url":"https://github.com/hejunqing/never-lost-in-the-middle/blob/HEAD/LongBench/src/metrics.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e7e75981cb464788"}},{"code_sha256_prefix":"8c5c581f9264c810","entry":"normalize_zh_answer","repo":"hejunqing/never-lost-in-the-middle","repo_kind":"official","path":"LongBench/src/metrics.py","file_url":"https://github.com/hejunqing/never-lost-in-the-middle/blob/HEAD/LongBench/src/metrics.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"8c5c581f9264c810"}},{"code_sha256_prefix":"36753478d8bfe1e1","entry":"parse_args","repo":"hejunqing/never-lost-in-the-middle","repo_kind":"official","path":"LongBench/src/pred_chatglm.py","file_url":"https://github.com/hejunqing/never-lost-in-the-middle/blob/HEAD/LongBench/src/pred_chatglm.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"36753478d8bfe1e1"}},{"code_sha256_prefix":"4489113b536ca6eb","entry":"post_process","repo":"hejunqing/never-lost-in-the-middle","repo_kind":"official","path":"LongBench/src/pred_chatglm.py","file_url":"https://github.com/hejunqing/never-lost-in-the-middle/blob/HEAD/LongBench/src/pred_chatglm.py","link_basis":"harvester_set","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"4489113b536ca6eb"}},{"code_sha256_prefix":"8b733c9fc590e4c1","entry":"preprocess","repo":"qihoo360/360zhinao","repo_kind":"listed","path":"Reranking/finetune.py","file_url":"https://github.com/qihoo360/360zhinao/blob/HEAD/Reranking/finetune.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"8b733c9fc590e4c1"}},{"code_sha256_prefix":"372ffbf8201660a5","entry":"replace_head","repo":"hejunqing/never-lost-in-the-middle","repo_kind":"official","path":"LongBench/src/new_utils.py","file_url":"https://github.com/hejunqing/never-lost-in-the-middle/blob/HEAD/LongBench/src/new_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"372ffbf8201660a5"}},{"code_sha256_prefix":"6653c42aad855a65","entry":"shorten","repo":"hejunqing/never-lost-in-the-middle","repo_kind":"official","path":"LongBench/src/new_utils.py","file_url":"https://github.com/hejunqing/never-lost-in-the-middle/blob/HEAD/LongBench/src/new_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"6653c42aad855a65"}},{"code_sha256_prefix":"0af745ab8b003692","entry":"build_chat","repo":"hejunqing/never-lost-in-the-middle","repo_kind":"official","path":"LongBench/src/pred_shuffle_reader.py","file_url":"https://github.com/hejunqing/never-lost-in-the-middle/blob/HEAD/LongBench/src/pred_shuffle_reader.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"0af745ab8b003692"}},{"code_sha256_prefix":"a30e2612231ea545","entry":"preprocess","repo":"qihoo360/360zhinao","repo_kind":"listed","path":"Reranking/flag_models.py","file_url":"https://github.com/qihoo360/360zhinao/blob/HEAD/Reranking/flag_models.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"a30e2612231ea545"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}