{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/arxiv-2601-07894","title":"Revealing the Attention Floating Mechanism in Masked Diffusion Models","arxiv_id":"2601.07894","date":"2026-01-12","proceeding":null,"authors":["Xin Dai","Pengcheng Huang","Zhenghao Liu","Shuo Wang","Yukun Yan","Chaojun Xiao","Yu Gu","Ge Yu","Maosong Sun"],"abstract":"Masked diffusion models (MDMs), which leverage bidirectional attention and a denoising process, are narrowing the performance gap with autoregressive models (ARMs). However, their internal attention mechanisms remain under-explored. This paper investigates the attention behaviors in MDMs, revealing the phenomenon of Attention Floating. Unlike ARMs, where attention converges to a fixed sink, MDMs exhibit dynamic, dispersed attention anchors that shift across denoising steps and layers. Further analysis reveals its Shallow Structure-Aware, Deep Content-Focused attention mechanism: shallow layers utilize floating tokens to build a global structural framework, while deeper layers allocate more capability toward capturing semantic content. Empirically, this distinctive attention pattern provides a mechanistic explanation for the strong in-context learning capabilities of MDMs, allowing them to double the performance compared to ARMs in knowledge-intensive tasks. All codes and datasets are available at https://github.com/NEUIR/Attention-Floating.","url_abs":"https://arxiv.org/abs/2601.07894","url_pdf":"https://arxiv.org/pdf/2601.07894","source":{"archive":null,"snapshot":"2025-07-28","note":"not in the Papers with Code archive (frozen at the snapshot)","row_kind":"graph","title_abstract_authors_date":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)"},"code_links":[],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2601.07894","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2601.07894"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"mentioned_in_github":null,"is_official":null,"provenance":"deterministic:regex_extraction","mentioned_in_paper":null,"url":"https://github.com/NEUIR/Attention-Floating","reach":{"status":"ok"}}],"summary":{"unverified":13},"by_repo_kind":{"found_in_text":{"samples":13,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":13,"samples":[{"code_sha256_prefix":"6a4180528ef351f2","entry":"accuracy_score","repo":"NEUIR/Attention-Floating","repo_kind":"found_in_text","path":"evaluate/evaluate_dream.py","file_url":"https://github.com/NEUIR/Attention-Floating/blob/HEAD/evaluate/evaluate_dream.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"6a4180528ef351f2"}},{"code_sha256_prefix":"2340e6453f04571f","entry":"add_gumbel_noise","repo":"NEUIR/Attention-Floating","repo_kind":"found_in_text","path":"src/attention_extraction/Llada.py","file_url":"https://github.com/NEUIR/Attention-Floating/blob/HEAD/src/attention_extraction/Llada.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"2340e6453f04571f"}},{"code_sha256_prefix":"f1db726d53551dd6","entry":"attention_rollout_region","repo":"NEUIR/Attention-Floating","repo_kind":"found_in_text","path":"src/visualization/attention_rollout.py","file_url":"https://github.com/NEUIR/Attention-Floating/blob/HEAD/src/visualization/attention_rollout.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"f1db726d53551dd6"}},{"code_sha256_prefix":"38d374516cad7ce5","entry":"batch_analyze_ar","repo":"NEUIR/Attention-Floating","repo_kind":"found_in_text","path":"src/visualization/absorption_comparison.py","file_url":"https://github.com/NEUIR/Attention-Floating/blob/HEAD/src/visualization/absorption_comparison.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"38d374516cad7ce5"}},{"code_sha256_prefix":"d4491256869f88bb","entry":"batch_analyze_dream","repo":"NEUIR/Attention-Floating","repo_kind":"found_in_text","path":"src/visualization/absorption_comparison.py","file_url":"https://github.com/NEUIR/Attention-Floating/blob/HEAD/src/visualization/absorption_comparison.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"d4491256869f88bb"}},{"code_sha256_prefix":"eb1895b1ae5f22aa","entry":"batch_analyze_llada","repo":"NEUIR/Attention-Floating","repo_kind":"found_in_text","path":"src/visualization/absorption_comparison.py","file_url":"https://github.com/NEUIR/Attention-Floating/blob/HEAD/src/visualization/absorption_comparison.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"eb1895b1ae5f22aa"}},{"code_sha256_prefix":"d230ff14a05ed8e6","entry":"compute_qk_cos_normprod","repo":"NEUIR/Attention-Floating","repo_kind":"found_in_text","path":"src/visualization/QK_decomposition.py","file_url":"https://github.com/NEUIR/Attention-Floating/blob/HEAD/src/visualization/QK_decomposition.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"d230ff14a05ed8e6"}},{"code_sha256_prefix":"b86a2fefc81c48b6","entry":"detect_model_type","repo":"NEUIR/Attention-Floating","repo_kind":"found_in_text","path":"src/visualization/QK_decomposition.py","file_url":"https://github.com/NEUIR/Attention-Floating/blob/HEAD/src/visualization/QK_decomposition.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"b86a2fefc81c48b6"}},{"code_sha256_prefix":"cb0727ddfdbb00d2","entry":"f1_score","repo":"NEUIR/Attention-Floating","repo_kind":"found_in_text","path":"evaluate/evaluate_dream.py","file_url":"https://github.com/NEUIR/Attention-Floating/blob/HEAD/evaluate/evaluate_dream.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"cb0727ddfdbb00d2"}},{"code_sha256_prefix":"6d8719bde8059d0a","entry":"find_sinks_from_attention","repo":"NEUIR/Attention-Floating","repo_kind":"found_in_text","path":"src/visualization/QK_decomposition.py","file_url":"https://github.com/NEUIR/Attention-Floating/blob/HEAD/src/visualization/QK_decomposition.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"6d8719bde8059d0a"}},{"code_sha256_prefix":"623e32f834b5c999","entry":"get_num_transfer_tokens","repo":"NEUIR/Attention-Floating","repo_kind":"found_in_text","path":"src/attention_extraction/Llada.py","file_url":"https://github.com/NEUIR/Attention-Floating/blob/HEAD/src/attention_extraction/Llada.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"623e32f834b5c999"}},{"code_sha256_prefix":"ff3b542378f5d19c","entry":"normalize_answer","repo":"NEUIR/Attention-Floating","repo_kind":"found_in_text","path":"evaluate/evaluate_dream.py","file_url":"https://github.com/NEUIR/Attention-Floating/blob/HEAD/evaluate/evaluate_dream.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"ff3b542378f5d19c"}},{"code_sha256_prefix":"3fff7d663b106945","entry":"visualize_rollout_attention","repo":"NEUIR/Attention-Floating","repo_kind":"found_in_text","path":"src/visualization/attention_rollout.py","file_url":"https://github.com/NEUIR/Attention-Floating/blob/HEAD/src/visualization/attention_rollout.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"3fff7d663b106945"}}]},"arxiv_metadata":{"licence":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)","fields":["title","abstract","authors","date"],"primary_category":"cs.LG","source":"arxiv_2026.jsonl"},"syntology_extracted_results":null}