{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/arxiv-2509-15235","title":"ViSpec: Accelerating Vision-Language Models with Vision-Aware Speculative Decoding","arxiv_id":"2509.15235","date":"2025-09-17","proceeding":"NeurIPS","authors":["Jialiang Kang","Han Shu","Wenshuo Li","Yingjie Zhai","Xinghao Chen"],"abstract":"Speculative decoding is a widely adopted technique for accelerating inference in large language models (LLMs), yet its application to vision-language models (VLMs) remains underexplored, with existing methods achieving only modest speedups (<1.5x). This gap is increasingly significant as multimodal capabilities become central to large-scale models. We hypothesize that large VLMs can effectively filter redundant image information layer by layer without compromising textual comprehension, whereas smaller draft models struggle to do so. To address this, we introduce Vision-Aware Speculative Decoding (ViSpec), a novel framework tailored for VLMs. ViSpec employs a lightweight vision adaptor module to compress image tokens into a compact representation, which is seamlessly integrated into the draft model's attention mechanism while preserving original image positional information. Additionally, we extract a global feature vector for each input image and augment all subsequent text tokens with this feature to enhance multimodal coherence. To overcome the scarcity of multimodal datasets with long assistant responses, we curate a specialized training dataset by repurposing existing datasets and generating extended outputs using the target VLM with modified prompts. Our training strategy mitigates the risk of the draft model exploiting direct access to the target model's hidden states, which could otherwise lead to shortcut learning when training solely on target model outputs. Extensive experiments validate ViSpec, achieving, to our knowledge, the first substantial speedup in VLM speculative decoding. Code is available at https://github.com/KangJialiang/ViSpec.","url_abs":"https://arxiv.org/abs/2509.15235","url_pdf":"https://arxiv.org/pdf/2509.15235","source":{"archive":null,"snapshot":"2025-07-28","note":"not in the Papers with Code archive (frozen at the snapshot)","row_kind":"graph","title_abstract_authors_date":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)"},"code_links":[],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2509.15235","atlas_url":"https://app.syntology.ai/?focus=2509.15235","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2509.15235"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"mentioned_in_github":null,"is_official":null,"provenance":"deterministic:regex_extraction","mentioned_in_paper":null,"url":"https://github.com/KangJialiang/ViSpec","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran_fixture":3,"ran":2,"ran_draft_wrong":2,"unverified":5},"by_repo_kind":{"found_in_text":{"samples":12,"ran":7,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"3c76e52815c5401d","entry":"repeat_kv","repo":"KangJialiang/ViSpec","repo_kind":"found_in_text","path":"vispec/model/cnets.py","file_url":"https://github.com/KangJialiang/ViSpec/blob/HEAD/vispec/model/cnets.py","link_basis":"harvester_set","language":"python","status":"ran_fixture","verification_level":2,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"3c76e52815c5401d"}},{"code_sha256_prefix":"d61c483a3c2b3156","entry":"apply_rotary_pos_emb","repo":"KangJialiang/ViSpec","repo_kind":"found_in_text","path":"vispec/model/modeling_mixtral_kv.py","file_url":"https://github.com/KangJialiang/ViSpec/blob/HEAD/vispec/model/modeling_mixtral_kv.py","link_basis":"harvester_set","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"d61c483a3c2b3156"}},{"code_sha256_prefix":"9a65a30d006fc96e","entry":"apply_rotary_pos_emb","repo":"KangJialiang/ViSpec","repo_kind":"found_in_text","path":"vispec/model/cnets.py","file_url":"https://github.com/KangJialiang/ViSpec/blob/HEAD/vispec/model/cnets.py","link_basis":"harvester_set","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"9a65a30d006fc96e"}},{"code_sha256_prefix":"f6e0c0fc3868b787","entry":"repeat_kv","repo":"KangJialiang/ViSpec","repo_kind":"found_in_text","path":"vispec/model/modeling_llama_kv.py","file_url":"https://github.com/KangJialiang/ViSpec/blob/HEAD/vispec/model/modeling_llama_kv.py","link_basis":"plan_row","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"f6e0c0fc3868b787"}},{"code_sha256_prefix":"b99eea6376d1e212","entry":"rotate_half","repo":"KangJialiang/ViSpec","repo_kind":"found_in_text","path":"vispec/model/modeling_mixtral_kv.py","file_url":"https://github.com/KangJialiang/ViSpec/blob/HEAD/vispec/model/modeling_mixtral_kv.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"b99eea6376d1e212"}},{"code_sha256_prefix":"e03d53ba9d4f9ae5","entry":"rotate_half","repo":"KangJialiang/ViSpec","repo_kind":"found_in_text","path":"vispec/model/cnets.py","file_url":"https://github.com/KangJialiang/ViSpec/blob/HEAD/vispec/model/cnets.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"e03d53ba9d4f9ae5"}},{"code_sha256_prefix":"b71b0ed70aa2939c","entry":"rotate_half","repo":"KangJialiang/ViSpec","repo_kind":"found_in_text","path":"vispec/model/modeling_llama_kv.py","file_url":"https://github.com/KangJialiang/ViSpec/blob/HEAD/vispec/model/modeling_llama_kv.py","link_basis":"plan_row","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"b71b0ed70aa2939c"}},{"code_sha256_prefix":"bad260f6d71db00c","entry":"apply_rotary_pos_emb","repo":"KangJialiang/ViSpec","repo_kind":"found_in_text","path":"vispec/model/modeling_llama_kv.py","file_url":"https://github.com/KangJialiang/ViSpec/blob/HEAD/vispec/model/modeling_llama_kv.py","link_basis":"plan_row","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"bad260f6d71db00c"}},{"code_sha256_prefix":"f4a61d6313f698ac","entry":"apply_rotary_pos_emb_flashatt","repo":"KangJialiang/ViSpec","repo_kind":"found_in_text","path":"vispec/model/modeling_qwen2_5_vl_kv.py","file_url":"https://github.com/KangJialiang/ViSpec/blob/HEAD/vispec/model/modeling_qwen2_5_vl_kv.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"f4a61d6313f698ac"}},{"code_sha256_prefix":"50a09accb6ca2549","entry":"apply_rotary_pos_emb_vision","repo":"KangJialiang/ViSpec","repo_kind":"found_in_text","path":"vispec/model/modeling_qwen2_5_vl_kv.py","file_url":"https://github.com/KangJialiang/ViSpec/blob/HEAD/vispec/model/modeling_qwen2_5_vl_kv.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"50a09accb6ca2549"}},{"code_sha256_prefix":"86a9a36211cc9bb5","entry":"initialize_past_key_values","repo":"KangJialiang/ViSpec","repo_kind":"found_in_text","path":"vispec/model/kv_cache.py","file_url":"https://github.com/KangJialiang/ViSpec/blob/HEAD/vispec/model/kv_cache.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"86a9a36211cc9bb5"}},{"code_sha256_prefix":"d9308d9bf18cd072","entry":"load_balancing_loss_func","repo":"KangJialiang/ViSpec","repo_kind":"found_in_text","path":"vispec/model/modeling_mixtral_kv.py","file_url":"https://github.com/KangJialiang/ViSpec/blob/HEAD/vispec/model/modeling_mixtral_kv.py","link_basis":"plan_row","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"d9308d9bf18cd072"}}]},"arxiv_metadata":{"licence":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)","fields":["title","abstract","authors","date"],"primary_category":"cs.CV","source":"arxiv_api"},"syntology_extracted_results":null}