{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/arxiv-2607-03876","title":"AdaptiveSD A Stability-Aware, Runtime-Adaptive Speculative Decoding Framework with Multi-Policy Orchestration for CPU-Constrained LLM Inference","arxiv_id":"2607.03876","date":"2026-07-04","proceeding":null,"authors":["Sadra Saremi"],"abstract":"With the rise of small quantized GGUF-based language models and their increasing use for on-device inference tasks, we have seen the growing need for an approach capable of reliably delivering these models at scale even under severe memory bandwidth constraints such as those imposed by pure CPU implementations. Fixed-depth speculative decoding has emerged as one promising technique, but in practice, it often leads to performance degradation due to either bandwidth saturation, instability, or even catastrophic resource exhaustion resulting in system failure. To overcome this problem, we introduce AdaptiveSD, a fully runtime-adaptive speculative decoding framework aimed at ensuring robust, reliable execution across the spectrum of model types and workloads. Our solution consists of four tightly-coupled components working together in a continuous feedback loop: a Runtime Monitoring Engine tracking multiple signals relevant to ongoing computation, an Adaptive Draft Controller enforcing an eleven rule policy hierarchy prioritizing system resource preservation over raw draft count, a Dynamic Policy Engine employing a suite of heuristic and reinforcement learning techniques to dynamically modify policies depending upon workload behavior, and finally, a KV Cache Coordination Layer managing cache states with fine-grained control through INT8 shadow buffers and position aware evictions. While conventional approaches focus solely on maximizing throughput, we instead assess the effectiveness of our approach based on several key metrics including wasted drafted compute and inter-token latency dispersion alongside standard measures of speculative efficiency.","url_abs":"https://arxiv.org/abs/2607.03876","url_pdf":"https://arxiv.org/pdf/2607.03876","source":{"archive":null,"snapshot":"2025-07-28","note":"not in the Papers with Code archive (frozen at the snapshot)","row_kind":"graph","title_abstract_authors_date":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)"},"code_links":[],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":null,"mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2607.03876"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"mentioned_in_github":null,"is_official":null,"provenance":"deterministic:regex_extraction","mentioned_in_paper":null,"url":"https://github.com/sadrasa97/adaptive-speculate-decoding","reach":null}],"summary":{"ran":13,"unverified":7},"by_repo_kind":{"found_in_text":{"samples":20,"ran":13,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":20,"samples":[{"code_sha256_prefix":"07372437b17496a6","entry":"ControlDecision","repo":"sadrasa97/adaptive-speculate-decoding","repo_kind":"found_in_text","path":"policy_engine.py","file_url":"https://github.com/sadrasa97/adaptive-speculate-decoding/blob/HEAD/policy_engine.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"07372437b17496a6"}},{"code_sha256_prefix":"d0c5cce56dba408b","entry":"DraftControllerConfig","repo":"sadrasa97/adaptive-speculate-decoding","repo_kind":"found_in_text","path":"policy_engine.py","file_url":"https://github.com/sadrasa97/adaptive-speculate-decoding/blob/HEAD/policy_engine.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"d0c5cce56dba408b"}},{"code_sha256_prefix":"c9bbee7cb7c9360e","entry":"DynamicLookaheadPolicy","repo":"sadrasa97/adaptive-speculate-decoding","repo_kind":"found_in_text","path":"policy_engine.py","file_url":"https://github.com/sadrasa97/adaptive-speculate-decoding/blob/HEAD/policy_engine.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"c9bbee7cb7c9360e"}},{"code_sha256_prefix":"77d690704bc16784","entry":"EMAPolicy","repo":"sadrasa97/adaptive-speculate-decoding","repo_kind":"found_in_text","path":"policy_engine.py","file_url":"https://github.com/sadrasa97/adaptive-speculate-decoding/blob/HEAD/policy_engine.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"77d690704bc16784"}},{"code_sha256_prefix":"e1cfcc9de9793be0","entry":"FixedDepthPolicy","repo":"sadrasa97/adaptive-speculate-decoding","repo_kind":"found_in_text","path":"policy_engine.py","file_url":"https://github.com/sadrasa97/adaptive-speculate-decoding/blob/HEAD/policy_engine.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"e1cfcc9de9793be0"}},{"code_sha256_prefix":"5403d7ca6666241b","entry":"InferenceSnapshot","repo":"sadrasa97/adaptive-speculate-decoding","repo_kind":"found_in_text","path":"policy_engine.py","file_url":"https://github.com/sadrasa97/adaptive-speculate-decoding/blob/HEAD/policy_engine.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"5403d7ca6666241b"}},{"code_sha256_prefix":"31d4395a3278c8c9","entry":"MonitorConfig","repo":"sadrasa97/adaptive-speculate-decoding","repo_kind":"found_in_text","path":"policy_engine.py","file_url":"https://github.com/sadrasa97/adaptive-speculate-decoding/blob/HEAD/policy_engine.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"31d4395a3278c8c9"}},{"code_sha256_prefix":"206911d54a5cf1e0","entry":"ParallelSDPolicy","repo":"sadrasa97/adaptive-speculate-decoding","repo_kind":"found_in_text","path":"policy_engine.py","file_url":"https://github.com/sadrasa97/adaptive-speculate-decoding/blob/HEAD/policy_engine.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"206911d54a5cf1e0"}},{"code_sha256_prefix":"bcfffd30b5673f1f","entry":"PolicyConfig","repo":"sadrasa97/adaptive-speculate-decoding","repo_kind":"found_in_text","path":"policy_engine.py","file_url":"https://github.com/sadrasa97/adaptive-speculate-decoding/blob/HEAD/policy_engine.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"bcfffd30b5673f1f"}},{"code_sha256_prefix":"2890c774d90eb401","entry":"PolicyStep","repo":"sadrasa97/adaptive-speculate-decoding","repo_kind":"found_in_text","path":"policy_engine.py","file_url":"https://github.com/sadrasa97/adaptive-speculate-decoding/blob/HEAD/policy_engine.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"2890c774d90eb401"}},{"code_sha256_prefix":"204fec1478e473e8","entry":"RuntimeMonitor","repo":"sadrasa97/adaptive-speculate-decoding","repo_kind":"found_in_text","path":"policy_engine.py","file_url":"https://github.com/sadrasa97/adaptive-speculate-decoding/blob/HEAD/policy_engine.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"204fec1478e473e8"}},{"code_sha256_prefix":"fd0dc8d250343d0c","entry":"SpecDecPlusApproxPolicy","repo":"sadrasa97/adaptive-speculate-decoding","repo_kind":"found_in_text","path":"policy_engine.py","file_url":"https://github.com/sadrasa97/adaptive-speculate-decoding/blob/HEAD/policy_engine.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"fd0dc8d250343d0c"}},{"code_sha256_prefix":"c75015ee60041029","entry":"_BanditArm","repo":"sadrasa97/adaptive-speculate-decoding","repo_kind":"found_in_text","path":"policy_engine.py","file_url":"https://github.com/sadrasa97/adaptive-speculate-decoding/blob/HEAD/policy_engine.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"c75015ee60041029"}},{"code_sha256_prefix":"d2988c254ee68db3","entry":"AdaptiveDraftController","repo":"sadrasa97/adaptive-speculate-decoding","repo_kind":"found_in_text","path":"policy_engine.py","file_url":"https://github.com/sadrasa97/adaptive-speculate-decoding/blob/HEAD/policy_engine.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"d2988c254ee68db3"}},{"code_sha256_prefix":"36fc1efcc88923c9","entry":"BanditPolicy","repo":"sadrasa97/adaptive-speculate-decoding","repo_kind":"found_in_text","path":"policy_engine.py","file_url":"https://github.com/sadrasa97/adaptive-speculate-decoding/blob/HEAD/policy_engine.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"36fc1efcc88923c9"}},{"code_sha256_prefix":"a447d4845282ff58","entry":"DynamicPolicyEngine","repo":"sadrasa97/adaptive-speculate-decoding","repo_kind":"found_in_text","path":"policy_engine.py","file_url":"https://github.com/sadrasa97/adaptive-speculate-decoding/blob/HEAD/policy_engine.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"a447d4845282ff58"}},{"code_sha256_prefix":"23e1e629a59655a3","entry":"HeuristicPolicy","repo":"sadrasa97/adaptive-speculate-decoding","repo_kind":"found_in_text","path":"policy_engine.py","file_url":"https://github.com/sadrasa97/adaptive-speculate-decoding/blob/HEAD/policy_engine.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"23e1e629a59655a3"}},{"code_sha256_prefix":"7a894723af7b633e","entry":"PolicyType","repo":"sadrasa97/adaptive-speculate-decoding","repo_kind":"found_in_text","path":"policy_engine.py","file_url":"https://github.com/sadrasa97/adaptive-speculate-decoding/blob/HEAD/policy_engine.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"7a894723af7b633e"}},{"code_sha256_prefix":"4b05210c360696f2","entry":"SpeculationMode","repo":"sadrasa97/adaptive-speculate-decoding","repo_kind":"found_in_text","path":"policy_engine.py","file_url":"https://github.com/sadrasa97/adaptive-speculate-decoding/blob/HEAD/policy_engine.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"4b05210c360696f2"}},{"code_sha256_prefix":"c2a752cfa024efad","entry":"WorkloadType","repo":"sadrasa97/adaptive-speculate-decoding","repo_kind":"found_in_text","path":"policy_engine.py","file_url":"https://github.com/sadrasa97/adaptive-speculate-decoding/blob/HEAD/policy_engine.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"c2a752cfa024efad"}}]},"arxiv_metadata":{"licence":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)","fields":["title","abstract","authors","date"],"primary_category":"cs.LG","source":"arxiv_2026.jsonl"},"syntology_extracted_results":null}