{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/arxiv-2601-21709","title":"Why Attention Patterns Exist: A Unifying Temporal Perspective Analysis","arxiv_id":"2601.21709","date":"2026-01-29","proceeding":null,"authors":["Qingyue Yang","Jie Wang","Xing Li","Yinqi Bai","Xialiang Tong","Huiling Zhen","Jianye Hao","Mingxuan Yuan","Bin Li"],"abstract":"Attention patterns play a crucial role in both training and inference of large language models (LLMs). Prior works have identified individual patterns such as retrieval heads, sink heads, and diagonal traces, yet these observations remain fragmented and lack a unifying explanation. To bridge this gap, we introduce \\textbf{Temporal Attention Pattern Predictability Analysis (TAPPA), a unifying framework that explains diverse attention patterns by analyzing their underlying mathematical formulations} from a temporally continuous perspective. TAPPA both deepens the understanding of attention behavior and guides inference acceleration approaches. Specifically, TAPPA characterizes attention patterns as predictable patterns with clear regularities and unpredictable patterns that appear effectively random. Our analysis further reveals that this distinction can be explained by the degree of query self-similarity along the temporal dimension. Focusing on the predictable patterns, we further provide a detailed mathematical analysis of three representative cases through the joint effect of queries, keys, and Rotary Positional Embeddings (RoPE). We validate TAPPA by applying its insights to KV cache compression and LLM pruning tasks. Across these tasks, a simple metric motivated by TAPPA consistently improves performance over baseline methods. The code is available at https://github.com/MIRALab-USTC/LLM-TAPPA.","url_abs":"https://arxiv.org/abs/2601.21709","url_pdf":"https://arxiv.org/pdf/2601.21709","source":{"archive":null,"snapshot":"2025-07-28","note":"not in the Papers with Code archive (frozen at the snapshot)","row_kind":"graph","title_abstract_authors_date":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)"},"code_links":[],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2601.21709","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2601.21709"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"mentioned_in_github":null,"is_official":null,"provenance":"deterministic:regex_extraction","mentioned_in_paper":null,"url":"https://github.com/MIRALab-USTC/LLM-TAPPA","reach":null}],"summary":{"ran_draft_wrong":2,"unverified":3},"by_repo_kind":{"found_in_text":{"samples":5,"ran":2,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"518b8a174b0afc4e","entry":"calculate_entropy","repo":"MIRALab-USTC/LLM-TAPPA","repo_kind":"found_in_text","path":"KVCache/tappa/model/llama_forward.py","file_url":"https://github.com/MIRALab-USTC/LLM-TAPPA/blob/HEAD/KVCache/tappa/model/llama_forward.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"518b8a174b0afc4e"}},{"code_sha256_prefix":"2685491bc8254d65","entry":"compute_q_sim_torch","repo":"MIRALab-USTC/LLM-TAPPA","repo_kind":"found_in_text","path":"KVCache/tappa/model/llama_forward.py","file_url":"https://github.com/MIRALab-USTC/LLM-TAPPA/blob/HEAD/KVCache/tappa/model/llama_forward.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"2685491bc8254d65"}},{"code_sha256_prefix":"645fbd38a3691771","entry":"CakeCache","repo":"MIRALab-USTC/LLM-TAPPA","repo_kind":"found_in_text","path":"KVCache/tappa/model/llama_forward.py","file_url":"https://github.com/MIRALab-USTC/LLM-TAPPA/blob/HEAD/KVCache/tappa/model/llama_forward.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"645fbd38a3691771"}},{"code_sha256_prefix":"96d4e192f7d7360e","entry":"CakeDecodingKVCache_LayerWise","repo":"MIRALab-USTC/LLM-TAPPA","repo_kind":"found_in_text","path":"KVCache/tappa/model/llama_forward.py","file_url":"https://github.com/MIRALab-USTC/LLM-TAPPA/blob/HEAD/KVCache/tappa/model/llama_forward.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"96d4e192f7d7360e"}},{"code_sha256_prefix":"94b5083393355144","entry":"llama_attn_forward_tappa","repo":"MIRALab-USTC/LLM-TAPPA","repo_kind":"found_in_text","path":"KVCache/tappa/model/llama_forward.py","file_url":"https://github.com/MIRALab-USTC/LLM-TAPPA/blob/HEAD/KVCache/tappa/model/llama_forward.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"94b5083393355144"}}]},"arxiv_metadata":{"licence":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)","fields":["title","abstract","authors","date"],"primary_category":"cs.CL","source":"arxiv_2026.jsonl"},"syntology_extracted_results":null}