{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/arxiv-2601-20520","title":"Context Tokens are Anchors: Understanding the Repetition Curse in dMLLMs from an Information Flow Perspective","arxiv_id":"2601.20520","date":"2026-01-28","proceeding":null,"authors":["Qiyan Zhao","Xiaofeng Zhang","Shuochen Chang","Qianyu Chen","Xiaosong Yuan","Xuhang Chen","Luoqi Liu","Jiajun Zhang","Xu-Yao Zhang","Da-Han Wang"],"abstract":"Recent diffusion-based Multimodal Large Language Models (dMLLMs) suffer from high inference latency and therefore rely on caching techniques to accelerate decoding. However, the application of cache mechanisms often introduces undesirable repetitive text generation, a phenomenon we term the \\textbf{Repeat Curse}. To better investigate underlying mechanism behind this issue, we analyze repetition generation through the lens of information flow. Our work reveals three key findings: (1) context tokens aggregate semantic information as anchors and guide the final predictions; (2) as information propagates across layers, the entropy of context tokens converges in deeper layers, reflecting the model's growing prediction certainty; (3) Repetition is typically linked to disruptions in the information flow of context tokens and to the inability of their entropy to converge in deeper layers. Based on these insights, we present \\textbf{CoTA}, a plug-and-play method for mitigating repetition. CoTA enhances the attention of context tokens to preserve intrinsic information flow patterns, while introducing a penalty term to the confidence score during decoding to avoid outputs driven by uncertain context tokens. With extensive experiments, CoTA demonstrates significant effectiveness in alleviating repetition and achieves consistent performance improvements on general tasks. Code is available at https://github.com/ErikZ719/CoTA","url_abs":"https://arxiv.org/abs/2601.20520","url_pdf":"https://arxiv.org/pdf/2601.20520","source":{"archive":null,"snapshot":"2025-07-28","note":"not in the Papers with Code archive (frozen at the snapshot)","row_kind":"graph","title_abstract_authors_date":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)"},"code_links":[],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":null,"atlas_url":"https://app.syntology.ai/?focus=2601.20520","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2601.20520"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"mentioned_in_github":null,"is_official":null,"provenance":"deterministic:regex_extraction","mentioned_in_paper":null,"url":"https://github.com/ErikZ719/CoTA","reach":null}],"summary":{"ran_draft_wrong":3,"unverified":4},"by_repo_kind":{"found_in_text":{"samples":7,"ran":3,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":7,"samples":[{"code_sha256_prefix":"62dcd26eb49d22ca","entry":"_dev","repo":"ErikZ719/CoTA","repo_kind":"found_in_text","path":"train/llava/hooks/cache_hook_LLaDA_V.py","file_url":"https://github.com/ErikZ719/CoTA/blob/HEAD/train/llava/hooks/cache_hook_LLaDA_V.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"62dcd26eb49d22ca"}},{"code_sha256_prefix":"e6e7e3b4a554a78b","entry":"_empty_on","repo":"ErikZ719/CoTA","repo_kind":"found_in_text","path":"train/llava/hooks/cache_hook_LLaDA_V.py","file_url":"https://github.com/ErikZ719/CoTA/blob/HEAD/train/llava/hooks/cache_hook_LLaDA_V.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"e6e7e3b4a554a78b"}},{"code_sha256_prefix":"384d407570646e54","entry":"_to_device","repo":"ErikZ719/CoTA","repo_kind":"found_in_text","path":"train/llava/hooks/cache_hook_LLaDA_V.py","file_url":"https://github.com/ErikZ719/CoTA/blob/HEAD/train/llava/hooks/cache_hook_LLaDA_V.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"384d407570646e54"}},{"code_sha256_prefix":"215caba8ede6dc93","entry":"Singleton","repo":"ErikZ719/CoTA","repo_kind":"found_in_text","path":"train/llava/hooks/cache_hook_LLaDA_V.py","file_url":"https://github.com/ErikZ719/CoTA/blob/HEAD/train/llava/hooks/cache_hook_LLaDA_V.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"215caba8ede6dc93"}},{"code_sha256_prefix":"e803b8482a068abf","entry":"dLLMCache","repo":"ErikZ719/CoTA","repo_kind":"found_in_text","path":"train/llava/hooks/cache_hook_LLaDA_V.py","file_url":"https://github.com/ErikZ719/CoTA/blob/HEAD/train/llava/hooks/cache_hook_LLaDA_V.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"e803b8482a068abf"}},{"code_sha256_prefix":"2b19b6ae3193c278","entry":"llada_cache_hook_feature","repo":"ErikZ719/CoTA","repo_kind":"found_in_text","path":"train/llava/hooks/cache_hook_LLaDA_V.py","file_url":"https://github.com/ErikZ719/CoTA/blob/HEAD/train/llava/hooks/cache_hook_LLaDA_V.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"2b19b6ae3193c278"}},{"code_sha256_prefix":"caa92e6fb44ab61f","entry":"refresh_index","repo":"ErikZ719/CoTA","repo_kind":"found_in_text","path":"train/llava/hooks/cache_hook_LLaDA_V.py","file_url":"https://github.com/ErikZ719/CoTA/blob/HEAD/train/llava/hooks/cache_hook_LLaDA_V.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"caa92e6fb44ab61f"}}]},"arxiv_metadata":{"licence":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)","fields":["title","abstract","authors","date"],"primary_category":"cs.CV","source":"arxiv_2026.jsonl"},"syntology_extracted_results":null}