{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/arxiv-2601-15498","title":"MARS: Unleashing the Power of Speculative Decoding via Margin-Aware Verification","arxiv_id":"2601.15498","date":"2026-01-21","proceeding":null,"authors":["Jingwei Song","Xinyu Wang","Hanbin Wang","Xiaoxuan Lei","Bill Shi","Shixin Han","Eric Yang","Xiao-Wen Chang","Lynn Ai"],"abstract":"Speculative Decoding (SD) accelerates autoregressive large language model (LLM) inference by decoupling generation and verification. While recent methods improve draft quality by tightly coupling the drafter with the target model, the verification mechanism itself remains largely unchanged, relying on strict token-level rejection sampling. In practice, modern LLMs frequently operate in low-margin regimes where the target model exhibits weak preference among top candidates. In such cases, rejecting plausible runner-up tokens yields negligible information gain while incurring substantial rollback cost, leading to a fundamental inefficiency in verification. We propose Margin-Aware Speculative Verification, a training-free and domain-agnostic verification strategy that adapts to the target model's local decisiveness. Our method conditions verification on decision stability measured directly from the target logits and relaxes rejection only when strict verification provides minimal benefit. Importantly, the approach modifies only the verification rule and is fully compatible with existing target-coupled speculative decoding frameworks. Extensive experiments across model scales ranging from 8B to 235B demonstrate that our method delivers consistent and significant inference speedups over state-of-the-art baselines while preserving generation quality across diverse benchmarks. The code is available at https://github.com/5SSjw/MARS.","url_abs":"https://arxiv.org/abs/2601.15498","url_pdf":"https://arxiv.org/pdf/2601.15498","source":{"archive":null,"snapshot":"2025-07-28","note":"not in the Papers with Code archive (frozen at the snapshot)","row_kind":"graph","title_abstract_authors_date":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)"},"code_links":[],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2601.15498","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2601.15498"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"mentioned_in_github":null,"is_official":null,"provenance":"deterministic:regex_extraction","mentioned_in_paper":null,"url":"https://github.com/5SSjw/MARS","reach":null}],"summary":{"ran_fixture":4,"ran":2,"ran_draft_wrong":2,"unverified":7},"by_repo_kind":{"found_in_text":{"samples":15,"ran":8,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"30d7eec482ebf6b1","entry":"repeat_kv","repo":"5SSjw/MARS","repo_kind":"found_in_text","path":"mars/model/cnets1.py","file_url":"https://github.com/5SSjw/MARS/blob/HEAD/mars/model/cnets1.py","link_basis":"harvester_set","language":"python","status":"ran_fixture","verification_level":2,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"30d7eec482ebf6b1"}},{"code_sha256_prefix":"3c76e52815c5401d","entry":"repeat_kv","repo":"5SSjw/MARS","repo_kind":"found_in_text","path":"mars/model/cnets.py","file_url":"https://github.com/5SSjw/MARS/blob/HEAD/mars/model/cnets.py","link_basis":"harvester_set","language":"python","status":"ran_fixture","verification_level":2,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"3c76e52815c5401d"}},{"code_sha256_prefix":"f725bc2d76076485","entry":"apply_rotary_pos_emb","repo":"5SSjw/MARS","repo_kind":"found_in_text","path":"mars/model/cnets1.py","file_url":"https://github.com/5SSjw/MARS/blob/HEAD/mars/model/cnets1.py","link_basis":"harvester_set","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f725bc2d76076485"}},{"code_sha256_prefix":"9a65a30d006fc96e","entry":"apply_rotary_pos_emb","repo":"5SSjw/MARS","repo_kind":"found_in_text","path":"mars/model/cnets.py","file_url":"https://github.com/5SSjw/MARS/blob/HEAD/mars/model/cnets.py","link_basis":"harvester_set","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"9a65a30d006fc96e"}},{"code_sha256_prefix":"a40c43d3dbc369d3","entry":"pad_path","repo":"5SSjw/MARS","repo_kind":"found_in_text","path":"mars/model/utils.py","file_url":"https://github.com/5SSjw/MARS/blob/HEAD/mars/model/utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a40c43d3dbc369d3"}},{"code_sha256_prefix":"78ae913ae6243c38","entry":"prepare_logits_processor","repo":"5SSjw/MARS","repo_kind":"found_in_text","path":"mars/model/utils.py","file_url":"https://github.com/5SSjw/MARS/blob/HEAD/mars/model/utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"78ae913ae6243c38"}},{"code_sha256_prefix":"b99eea6376d1e212","entry":"rotate_half","repo":"5SSjw/MARS","repo_kind":"found_in_text","path":"mars/model/cnets1.py","file_url":"https://github.com/5SSjw/MARS/blob/HEAD/mars/model/cnets1.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b99eea6376d1e212"}},{"code_sha256_prefix":"e03d53ba9d4f9ae5","entry":"rotate_half","repo":"5SSjw/MARS","repo_kind":"found_in_text","path":"mars/model/cnets.py","file_url":"https://github.com/5SSjw/MARS/blob/HEAD/mars/model/cnets.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e03d53ba9d4f9ae5"}},{"code_sha256_prefix":"eee34413e2bccab6","entry":"apply_rotary_pos_emb","repo":"5SSjw/MARS","repo_kind":"found_in_text","path":"mars/model/modeling_llama_kv.py","file_url":"https://github.com/5SSjw/MARS/blob/HEAD/mars/model/modeling_llama_kv.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"eee34413e2bccab6"}},{"code_sha256_prefix":"484407b65b6e2d25","entry":"apply_rotary_pos_emb_L31","repo":"5SSjw/MARS","repo_kind":"found_in_text","path":"mars/model/modeling_llama_kv.py","file_url":"https://github.com/5SSjw/MARS/blob/HEAD/mars/model/modeling_llama_kv.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"484407b65b6e2d25"}},{"code_sha256_prefix":"f64290cf90293f24","entry":"build_chat_messages","repo":"5SSjw/MARS","repo_kind":"found_in_text","path":"mars/eval/gen_baseline_answer_llama3chat.py","file_url":"https://github.com/5SSjw/MARS/blob/HEAD/mars/eval/gen_baseline_answer_llama3chat.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f64290cf90293f24"}},{"code_sha256_prefix":"bbfbabefe7ad3955","entry":"initialize_past_key_values","repo":"5SSjw/MARS","repo_kind":"found_in_text","path":"mars/model/kv_cache.py","file_url":"https://github.com/5SSjw/MARS/blob/HEAD/mars/model/kv_cache.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"bbfbabefe7ad3955"}},{"code_sha256_prefix":"d9308d9bf18cd072","entry":"load_balancing_loss_func","repo":"5SSjw/MARS","repo_kind":"found_in_text","path":"mars/model/modeling_mixtral_kv.py","file_url":"https://github.com/5SSjw/MARS/blob/HEAD/mars/model/modeling_mixtral_kv.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d9308d9bf18cd072"}},{"code_sha256_prefix":"ded2d9e57bfdee4e","entry":"rotate_half","repo":"5SSjw/MARS","repo_kind":"found_in_text","path":"mars/model/modeling_llama_kv.py","file_url":"https://github.com/5SSjw/MARS/blob/HEAD/mars/model/modeling_llama_kv.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ded2d9e57bfdee4e"}},{"code_sha256_prefix":"e0affdb39752f507","entry":"save_logits_distribution","repo":"5SSjw/MARS","repo_kind":"found_in_text","path":"mars/model/utils.py","file_url":"https://github.com/5SSjw/MARS/blob/HEAD/mars/model/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e0affdb39752f507"}}]},"arxiv_metadata":{"licence":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)","fields":["title","abstract","authors","date"],"primary_category":"cs.LG","source":"arxiv_2026.jsonl"},"syntology_extracted_results":null}