{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/distributed-speculative-inference-of-large","title":"Distributed Speculative Inference (DSI): Speculation Parallelism for Provably Faster Lossless Language Model Inference","arxiv_id":"2405.14105","date":"2024-05-23","proceeding":null,"authors":["Nadav Timor","Jonathan Mamou","Daniel Korat","Moshe Berchansky","Oren Pereg","Moshe Wasserblat","Tomer Galanti","Michal Gordon","David Harel"],"abstract":"This paper introduces distributed speculative inference (DSI), a novel inference algorithm that is provably faster than speculative inference (SI) [leviathan2023, chen2023, miao2024, sun2025, timor2025] and standard autoregressive inference (non-SI). Like other SI algorithms, DSI operates on frozen language models (LMs), requiring no training or architectural modifications, and it preserves the target distribution. Prior studies on SI have demonstrated empirical speedups over non-SI--but rely on sufficiently fast and accurate drafters, which are often unavailable in practice. We identify a gap where SI can be slower than non-SI if drafters are too slow or inaccurate. We close this gap by proving that DSI is faster than both SI and non-SI--given any drafters. DSI is therefore not only faster than SI, but also unlocks the acceleration of LMs for which SI fails. DSI leverages speculation parallelism (SP), a novel type of task parallelism, to orchestrate target and drafter instances that overlap in time, establishing a new foundational tradeoff between computational resources and latency. Our simulations show that DSI is 1.29-1.92x faster than SI in single-node setups for various off-the-shelf LMs and tasks. We open-source all our code.","url_abs":"https://arxiv.org/abs/2405.14105v5","url_pdf":"https://arxiv.org/pdf/2405.14105v5.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"distributed-speculative-inference-of-large","repo_url":"https://github.com/keyboardAnt/distributed-speculative-inference","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":null}],"tasks":[{"task_slug":"language-modeling","task_name":"Language Modeling"},{"task_slug":"language-modelling","task_name":"Language Modelling"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2405.14105","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2405.14105"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/keyboardAnt/distributed-speculative-inference","reach":null}],"summary":{"ran":6,"ran_draft_wrong":1,"ran_honours":1,"unverified":7},"by_repo_kind":{"official":{"samples":15,"ran":8,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"271a632f4512afd2","entry":"ConfigDSI","repo":"keyboardAnt/distributed-speculative-inference","repo_kind":"official","path":"dsi/online/simul/core.py","file_url":"https://github.com/keyboardAnt/distributed-speculative-inference/blob/HEAD/dsi/online/simul/core.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"271a632f4512afd2"}},{"code_sha256_prefix":"dacab08eab0d6559","entry":"ConfigSI","repo":"keyboardAnt/distributed-speculative-inference","repo_kind":"official","path":"dsi/online/simul/core.py","file_url":"https://github.com/keyboardAnt/distributed-speculative-inference/blob/HEAD/dsi/online/simul/core.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"dacab08eab0d6559"}},{"code_sha256_prefix":"4e59d13c19539020","entry":"DrafterSlowerThanTargetError","repo":"keyboardAnt/distributed-speculative-inference","repo_kind":"official","path":"dsi/online/simul/core.py","file_url":"https://github.com/keyboardAnt/distributed-speculative-inference/blob/HEAD/dsi/online/simul/core.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"4e59d13c19539020"}},{"code_sha256_prefix":"ec44f925c15ddb73","entry":"NumOfTargetServersInsufficientError","repo":"keyboardAnt/distributed-speculative-inference","repo_kind":"official","path":"dsi/online/simul/core.py","file_url":"https://github.com/keyboardAnt/distributed-speculative-inference/blob/HEAD/dsi/online/simul/core.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"ec44f925c15ddb73"}},{"code_sha256_prefix":"d8c80e158adc39c1","entry":"_ConfigExperiment","repo":"keyboardAnt/distributed-speculative-inference","repo_kind":"official","path":"dsi/online/simul/core.py","file_url":"https://github.com/keyboardAnt/distributed-speculative-inference/blob/HEAD/dsi/online/simul/core.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"d8c80e158adc39c1"}},{"code_sha256_prefix":"7945a734a6fcfc6f","entry":"call_target_actual","repo":"keyboardAnt/distributed-speculative-inference","repo_kind":"official","path":"dsi/online/simul/core.py","file_url":"https://github.com/keyboardAnt/distributed-speculative-inference/blob/HEAD/dsi/online/simul/core.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"7945a734a6fcfc6f"}},{"code_sha256_prefix":"7feffeae9b19b765","entry":"get_current_thread_name","repo":"keyboardAnt/distributed-speculative-inference","repo_kind":"official","path":"dsi/online/simul/core.py","file_url":"https://github.com/keyboardAnt/distributed-speculative-inference/blob/HEAD/dsi/online/simul/core.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"7feffeae9b19b765"}},{"code_sha256_prefix":"cc91710f73b2bd08","entry":"get_target_time","repo":"keyboardAnt/distributed-speculative-inference","repo_kind":"official","path":"dsi/online/simul/core.py","file_url":"https://github.com/keyboardAnt/distributed-speculative-inference/blob/HEAD/dsi/online/simul/core.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"cc91710f73b2bd08"}},{"code_sha256_prefix":"26701e3e474f9fef","entry":"ConfigDSIOnline","repo":"keyboardAnt/distributed-speculative-inference","repo_kind":"official","path":"dsi/online/simul/core.py","file_url":"https://github.com/keyboardAnt/distributed-speculative-inference/blob/HEAD/dsi/online/simul/core.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"26701e3e474f9fef"}},{"code_sha256_prefix":"b11aa6140555d2dc","entry":"SimulType","repo":"keyboardAnt/distributed-speculative-inference","repo_kind":"official","path":"dsi/online/simul/core.py","file_url":"https://github.com/keyboardAnt/distributed-speculative-inference/blob/HEAD/dsi/online/simul/core.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"b11aa6140555d2dc"}},{"code_sha256_prefix":"9ee7c56034b83c31","entry":"call_target","repo":"keyboardAnt/distributed-speculative-inference","repo_kind":"official","path":"dsi/online/simul/core.py","file_url":"https://github.com/keyboardAnt/distributed-speculative-inference/blob/HEAD/dsi/online/simul/core.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"9ee7c56034b83c31"}},{"code_sha256_prefix":"1514d09de3a360a6","entry":"fix_history","repo":"keyboardAnt/distributed-speculative-inference","repo_kind":"official","path":"dsi/online/simul/core.py","file_url":"https://github.com/keyboardAnt/distributed-speculative-inference/blob/HEAD/dsi/online/simul/core.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"1514d09de3a360a6"}},{"code_sha256_prefix":"753922cbd39c3d6d","entry":"run_generate","repo":"keyboardAnt/distributed-speculative-inference","repo_kind":"official","path":"dsi/online/simul/core.py","file_url":"https://github.com/keyboardAnt/distributed-speculative-inference/blob/HEAD/dsi/online/simul/core.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"753922cbd39c3d6d"}},{"code_sha256_prefix":"1ca3289bc87391f4","entry":"target_done_callback","repo":"keyboardAnt/distributed-speculative-inference","repo_kind":"official","path":"dsi/online/simul/core.py","file_url":"https://github.com/keyboardAnt/distributed-speculative-inference/blob/HEAD/dsi/online/simul/core.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"1ca3289bc87391f4"}},{"code_sha256_prefix":"a4e64bdf694d83db","entry":"terminate_process","repo":"keyboardAnt/distributed-speculative-inference","repo_kind":"official","path":"dsi/online/simul/core.py","file_url":"https://github.com/keyboardAnt/distributed-speculative-inference/blob/HEAD/dsi/online/simul/core.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"a4e64bdf694d83db"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}