{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/mitigate-position-bias-in-large-language","title":"Mitigate Position Bias in Large Language Models via Scaling a Single Dimension","arxiv_id":"2406.02536","date":"2024-06-04","proceeding":null,"authors":["Yijiong Yu","Huiqiang Jiang","Xufang Luo","Qianhui Wu","Chin-Yew Lin","Dongsheng Li","Yuqing Yang","Yongfeng Huang","Lili Qiu"],"abstract":"Large Language Models (LLMs) are increasingly applied in various real-world scenarios due to their excellent generalization capabilities and robust generative abilities. However, they exhibit position bias, also known as \"lost in the middle\", a phenomenon that is especially pronounced in long-context scenarios, which indicates the placement of the key information in different positions of a prompt can significantly affect accuracy. This paper first explores the micro-level manifestations of position bias, concluding that attention weights are a micro-level expression of position bias. It further identifies that, in addition to position embeddings, causal attention mask also contributes to position bias by creating position-specific hidden states. Based on these insights, we propose a method to mitigate position bias by scaling this positional hidden states. Experiments on the NaturalQuestions Multi-document QA, KV retrieval, LongBench and timeline reorder tasks, using various models including RoPE models, context windowextended models, and Alibi models, demonstrate the effectiveness and generalizability of our approach. Our method can improve performance by up to 15.2% by modifying just one dimension of hidden states. Our code is available at https://aka.ms/PositionalHidden.","url_abs":"https://arxiv.org/abs/2406.02536v2","url_pdf":"https://arxiv.org/pdf/2406.02536v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"mitigate-position-bias-in-large-language","repo_url":"https://github.com/PositionalHidden/PositionalHidden","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":null,"task_name":"Position"}],"methods":[{"method_slug":"alibi","method_name":"ALiBi"},{"method_slug":"rope","method_name":"Rotary Embeddings"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2406.02536","atlas_url":"https://app.syntology.ai/?focus=2406.02536","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2406.02536"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/PositionalHidden/PositionalHidden","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran_fixture":2,"ran_draft_wrong":4,"ran":10,"unverified":6},"by_repo_kind":{"official":{"samples":22,"ran":16,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"30d7eec482ebf6b1","entry":"repeat_kv","repo":"PositionalHidden/PositionalHidden","repo_kind":"official","path":"positional_hidden/models/modeling_gemma_hidden_scale.py","file_url":"https://github.com/PositionalHidden/PositionalHidden/blob/HEAD/positional_hidden/models/modeling_gemma_hidden_scale.py","link_basis":"harvester_set","language":"python","status":"ran_fixture","verification_level":2,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"30d7eec482ebf6b1"}},{"code_sha256_prefix":"d61c483a3c2b3156","entry":"apply_rotary_pos_emb","repo":"PositionalHidden/PositionalHidden","repo_kind":"official","path":"positional_hidden/models/modeling_llama_hidden_scale.py","file_url":"https://github.com/PositionalHidden/PositionalHidden/blob/HEAD/positional_hidden/models/modeling_llama_hidden_scale.py","link_basis":"harvester_set","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d61c483a3c2b3156"}},{"code_sha256_prefix":"bac65c3dafaec040","entry":"apply_rotary_pos_emb","repo":"PositionalHidden/PositionalHidden","repo_kind":"official","path":"positional_hidden/models/modeling_gemma_hidden_scale.py","file_url":"https://github.com/PositionalHidden/PositionalHidden/blob/HEAD/positional_hidden/models/modeling_gemma_hidden_scale.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"bac65c3dafaec040"}},{"code_sha256_prefix":"061f05dab3ffd652","entry":"apply_rotary_pos_emb_gemma","repo":"PositionalHidden/PositionalHidden","repo_kind":"official","path":"positional_hidden/models/modeling_llama_hidden_scale.py","file_url":"https://github.com/PositionalHidden/PositionalHidden/blob/HEAD/positional_hidden/models/modeling_llama_hidden_scale.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"061f05dab3ffd652"}},{"code_sha256_prefix":"10af7cdbc53df7e5","entry":"best_subspan_em","repo":"PositionalHidden/PositionalHidden","repo_kind":"official","path":"experiments/lost_in_the_middle/metrics.py","file_url":"https://github.com/PositionalHidden/PositionalHidden/blob/HEAD/experiments/lost_in_the_middle/metrics.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"10af7cdbc53df7e5"}},{"code_sha256_prefix":"6acdafe16c0d4786","entry":"build_mpt_alibi_tensor","repo":"PositionalHidden/PositionalHidden","repo_kind":"official","path":"positional_hidden/models/modeling_mpt_hidden_scale.py","file_url":"https://github.com/PositionalHidden/PositionalHidden/blob/HEAD/positional_hidden/models/modeling_mpt_hidden_scale.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"6acdafe16c0d4786"}},{"code_sha256_prefix":"1051445c95f4d2fc","entry":"count_score","repo":"PositionalHidden/PositionalHidden","repo_kind":"official","path":"experiments/run_on_longbench/metrics.py","file_url":"https://github.com/PositionalHidden/PositionalHidden/blob/HEAD/experiments/run_on_longbench/metrics.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"1051445c95f4d2fc"}},{"code_sha256_prefix":"76822e2ad71afee9","entry":"eval_dims","repo":"PositionalHidden/PositionalHidden","repo_kind":"official","path":"experiments/search_dimension/eval_on_valid_set.py","file_url":"https://github.com/PositionalHidden/PositionalHidden/blob/HEAD/experiments/search_dimension/eval_on_valid_set.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"76822e2ad71afee9"}},{"code_sha256_prefix":"b9ac8a016e78383a","entry":"format_chat_prompt","repo":"PositionalHidden/PositionalHidden","repo_kind":"official","path":"experiments/KV/run_hidden_scale_KV.py","file_url":"https://github.com/PositionalHidden/PositionalHidden/blob/HEAD/experiments/KV/run_hidden_scale_KV.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b9ac8a016e78383a"}},{"code_sha256_prefix":"6c56afd9bf4ea601","entry":"format_chat_prompt","repo":"PositionalHidden/PositionalHidden","repo_kind":"official","path":"experiments/run_on_longbench/run_on_longbench.py","file_url":"https://github.com/PositionalHidden/PositionalHidden/blob/HEAD/experiments/run_on_longbench/run_on_longbench.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"6c56afd9bf4ea601"}},{"code_sha256_prefix":"e52c1a2fa822f109","entry":"format_chat_prompt","repo":"PositionalHidden/PositionalHidden","repo_kind":"official","path":"experiments/search_dimension/eval_on_valid_set.py","file_url":"https://github.com/PositionalHidden/PositionalHidden/blob/HEAD/experiments/search_dimension/eval_on_valid_set.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e52c1a2fa822f109"}},{"code_sha256_prefix":"2b1ae840dfdd7e16","entry":"get_model_class","repo":"PositionalHidden/PositionalHidden","repo_kind":"official","path":"positional_hidden/use_model.py","file_url":"https://github.com/PositionalHidden/PositionalHidden/blob/HEAD/positional_hidden/use_model.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"2b1ae840dfdd7e16"}},{"code_sha256_prefix":"e7e75981cb464788","entry":"normalize_answer","repo":"PositionalHidden/PositionalHidden","repo_kind":"official","path":"experiments/run_on_longbench/metrics.py","file_url":"https://github.com/PositionalHidden/PositionalHidden/blob/HEAD/experiments/run_on_longbench/metrics.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e7e75981cb464788"}},{"code_sha256_prefix":"8c5c581f9264c810","entry":"normalize_zh_answer","repo":"PositionalHidden/PositionalHidden","repo_kind":"official","path":"experiments/run_on_longbench/metrics.py","file_url":"https://github.com/PositionalHidden/PositionalHidden/blob/HEAD/experiments/run_on_longbench/metrics.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"8c5c581f9264c810"}},{"code_sha256_prefix":"e60745782705e531","entry":"prompt_func","repo":"PositionalHidden/PositionalHidden","repo_kind":"official","path":"experiments/run_on_longbench/run_on_longbench.py","file_url":"https://github.com/PositionalHidden/PositionalHidden/blob/HEAD/experiments/run_on_longbench/run_on_longbench.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e60745782705e531"}},{"code_sha256_prefix":"b99eea6376d1e212","entry":"rotate_half","repo":"PositionalHidden/PositionalHidden","repo_kind":"official","path":"positional_hidden/models/modeling_gemma_hidden_scale.py","file_url":"https://github.com/PositionalHidden/PositionalHidden/blob/HEAD/positional_hidden/models/modeling_gemma_hidden_scale.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b99eea6376d1e212"}},{"code_sha256_prefix":"91506e645c7a9e50","entry":"get_closedbook_qa_prompt","repo":"PositionalHidden/PositionalHidden","repo_kind":"official","path":"experiments/lost_in_the_middle/prompting.py","file_url":"https://github.com/PositionalHidden/PositionalHidden/blob/HEAD/experiments/lost_in_the_middle/prompting.py","link_basis":"plan_row","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"91506e645c7a9e50"}},{"code_sha256_prefix":"6325cba09a03eb90","entry":"get_hidden_states","repo":"PositionalHidden/PositionalHidden","repo_kind":"official","path":"experiments/search_dimension/get_avg_hidden_state.py","file_url":"https://github.com/PositionalHidden/PositionalHidden/blob/HEAD/experiments/search_dimension/get_avg_hidden_state.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"6325cba09a03eb90"}},{"code_sha256_prefix":"aca69348624f921b","entry":"get_kv_retrieval_prompt","repo":"PositionalHidden/PositionalHidden","repo_kind":"official","path":"experiments/lost_in_the_middle/prompting.py","file_url":"https://github.com/PositionalHidden/PositionalHidden/blob/HEAD/experiments/lost_in_the_middle/prompting.py","link_basis":"plan_row","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"aca69348624f921b"}},{"code_sha256_prefix":"3fbc0e8f6cb70422","entry":"get_model_class","repo":"PositionalHidden/PositionalHidden","repo_kind":"official","path":"experiments/KV/run_hidden_scale_KV.py","file_url":"https://github.com/PositionalHidden/PositionalHidden/blob/HEAD/experiments/KV/run_hidden_scale_KV.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"3fbc0e8f6cb70422"}},{"code_sha256_prefix":"286b6ac3dc4a717d","entry":"get_qa_prompt","repo":"PositionalHidden/PositionalHidden","repo_kind":"official","path":"experiments/lost_in_the_middle/prompting.py","file_url":"https://github.com/PositionalHidden/PositionalHidden/blob/HEAD/experiments/lost_in_the_middle/prompting.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"286b6ac3dc4a717d"}},{"code_sha256_prefix":"9706a9a0b961096b","entry":"normalize_answer","repo":"PositionalHidden/PositionalHidden","repo_kind":"official","path":"experiments/lost_in_the_middle/metrics.py","file_url":"https://github.com/PositionalHidden/PositionalHidden/blob/HEAD/experiments/lost_in_the_middle/metrics.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"9706a9a0b961096b"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}