{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/efficient-inference-for-large-language-model","title":"Efficient Inference for Large Language Model-based Generative Recommendation","arxiv_id":"2410.05165","date":"2024-10-07","proceeding":null,"authors":["Xinyu Lin","Chaoqun Yang","Wenjie Wang","Yongqi Li","Cunxiao Du","Fuli Feng","See-Kiong Ng","Tat-Seng Chua"],"abstract":"Large Language Model (LLM)-based generative recommendation has achieved notable success, yet its practical deployment is costly particularly due to excessive inference latency caused by autoregressive decoding. For lossless LLM decoding acceleration, Speculative Decoding (SD) has emerged as a promising solution. However, applying SD to generative recommendation presents unique challenges due to the requirement of generating top-K items (i.e., K distinct token sequences) as a recommendation list by beam search. This leads to more stringent verification in SD, where all the top-K sequences from the target LLM must be successfully drafted by the draft model at each decoding step. To alleviate this, we consider 1) boosting top-K sequence alignment between the draft model and the target LLM, and 2) relaxing the verification strategy to reduce trivial LLM calls. To this end, we propose an alignment framework named AtSpeed, which presents the AtSpeed-S optimization objective for top-K alignment under the strict top-K verification. Moreover, we introduce a relaxed sampling verification strategy that allows high-probability non-top-K drafted sequences to be accepted, significantly reducing LLM calls. Correspondingly, we propose AtSpeed-R for top-K alignment under this relaxed sampling verification. Empirical results on two real-world datasets demonstrate that AtSpeed significantly accelerates LLM-based generative recommendation, e.g., near 2x speedup under strict top-K verification and up to 2.5x speedup under relaxed sampling verification. The codes and datasets are released at https://github.com/Linxyhaha/AtSpeed.","url_abs":"https://arxiv.org/abs/2410.05165v3","url_pdf":"https://arxiv.org/pdf/2410.05165v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"efficient-inference-for-large-language-model","repo_url":"https://github.com/linxyhaha/atspeed","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":"language-modeling","task_name":"Language Modeling"},{"task_slug":"language-modelling","task_name":"Language Modelling"},{"task_slug":"large-language-model","task_name":"Large Language Model"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2410.05165","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2410.05165"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/Linxyhaha/AtSpeed","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/linxyhaha/atspeed","reach":{"status":"ok"}}],"summary":{"ran":6,"ran_draft_wrong":1,"unverified":5},"by_repo_kind":{"official":{"samples":12,"ran":7,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":12,"samples":[{"code_sha256_prefix":"7dc4a09e7d23afd8","entry":"AtSpeedSModel","repo":"linxyhaha/atspeed","repo_kind":"official","path":"code/model.py","file_url":"https://github.com/linxyhaha/atspeed/blob/HEAD/code/model.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"7dc4a09e7d23afd8"}},{"code_sha256_prefix":"e0fcb137d04baf1f","entry":"bytes_to_gb","repo":"Linxyhaha/AtSpeed","repo_kind":"official","path":"code/generate_teacher_data.py","file_url":"https://github.com/Linxyhaha/AtSpeed/blob/HEAD/code/generate_teacher_data.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"e0fcb137d04baf1f"}},{"code_sha256_prefix":"daa3d7a4f1dbe7ae","entry":"get_prefix_allowed_tokens_fn","repo":"linxyhaha/atspeed","repo_kind":"official","path":"code/inference.py","file_url":"https://github.com/linxyhaha/atspeed/blob/HEAD/code/inference.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"daa3d7a4f1dbe7ae"}},{"code_sha256_prefix":"0345660faa8d6037","entry":"one_step_beam_search","repo":"Linxyhaha/AtSpeed","repo_kind":"official","path":"code/beamSD.py","file_url":"https://github.com/Linxyhaha/AtSpeed/blob/HEAD/code/beamSD.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"0345660faa8d6037"}},{"code_sha256_prefix":"fec89cf4d4ffc584","entry":"parse_dataset_args","repo":"Linxyhaha/AtSpeed","repo_kind":"official","path":"code/utils.py","file_url":"https://github.com/Linxyhaha/AtSpeed/blob/HEAD/code/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"fec89cf4d4ffc584"}},{"code_sha256_prefix":"38d59d157a39fe01","entry":"parse_global_args","repo":"Linxyhaha/AtSpeed","repo_kind":"official","path":"code/utils.py","file_url":"https://github.com/Linxyhaha/AtSpeed/blob/HEAD/code/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"38d59d157a39fe01"}},{"code_sha256_prefix":"3e27e1971d49a1a8","entry":"prefix_allowed_tokens_fn","repo":"Linxyhaha/AtSpeed","repo_kind":"official","path":"code/generation_trie.py","file_url":"https://github.com/Linxyhaha/AtSpeed/blob/HEAD/code/generation_trie.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"3e27e1971d49a1a8"}},{"code_sha256_prefix":"98dce01478979c79","entry":"BSSD","repo":"Linxyhaha/AtSpeed","repo_kind":"official","path":"code/beamSD.py","file_url":"https://github.com/Linxyhaha/AtSpeed/blob/HEAD/code/beamSD.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"98dce01478979c79"}},{"code_sha256_prefix":"31b0d67913003d72","entry":"generate_prompt","repo":"Linxyhaha/AtSpeed","repo_kind":"official","path":"code/finetune_llama.py","file_url":"https://github.com/Linxyhaha/AtSpeed/blob/HEAD/code/finetune_llama.py","link_basis":"plan_row","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"31b0d67913003d72"}},{"code_sha256_prefix":"e2f64861d2469ffb","entry":"main","repo":"Linxyhaha/AtSpeed","repo_kind":"official","path":"code/generate_teacher_data.py","file_url":"https://github.com/Linxyhaha/AtSpeed/blob/HEAD/code/generate_teacher_data.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"e2f64861d2469ffb"}},{"code_sha256_prefix":"f43271f389825ef1","entry":"parse_AtSpeed_args","repo":"Linxyhaha/AtSpeed","repo_kind":"official","path":"code/utils.py","file_url":"https://github.com/Linxyhaha/AtSpeed/blob/HEAD/code/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"f43271f389825ef1"}},{"code_sha256_prefix":"f10b46ee09ea6eb9","entry":"verify","repo":"Linxyhaha/AtSpeed","repo_kind":"official","path":"code/beamSD.py","file_url":"https://github.com/Linxyhaha/AtSpeed/blob/HEAD/code/beamSD.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"f10b46ee09ea6eb9"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}