{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/atom-low-bit-quantization-for-efficient-and","title":"Atom: Low-bit Quantization for Efficient and Accurate LLM Serving","arxiv_id":"2310.19102","date":"2023-10-29","proceeding":null,"authors":["Yilong Zhao","Chien-Yu Lin","Kan Zhu","Zihao Ye","Lequn Chen","Size Zheng","Luis Ceze","Arvind Krishnamurthy","Tianqi Chen","Baris Kasikci"],"abstract":"The growing demand for Large Language Models (LLMs) in applications such as content generation, intelligent chatbots, and sentiment analysis poses considerable challenges for LLM service providers. To efficiently use GPU resources and boost throughput, batching multiple requests has emerged as a popular paradigm; to further speed up batching, LLM quantization techniques reduce memory consumption and increase computing capacity. However, prevalent quantization schemes (e.g., 8-bit weight-activation quantization) cannot fully leverage the capabilities of modern GPUs, such as 4-bit integer operators, resulting in sub-optimal performance. To maximize LLMs' serving throughput, we introduce Atom, a low-bit quantization method that achieves high throughput improvements with negligible accuracy loss. Atom significantly boosts serving throughput by using low-bit operators and considerably reduces memory consumption via low-bit quantization. It attains high accuracy by applying a novel mixed-precision and fine-grained quantization process. We evaluate Atom on 4-bit weight-activation quantization in the serving context. Atom improves end-to-end throughput (token/s) by up to $7.7\\times$ compared to the FP16 and by $2.5\\times$ compared to INT8 quantization, while maintaining the same latency target.","url_abs":"https://arxiv.org/abs/2310.19102v3","url_pdf":"https://arxiv.org/pdf/2310.19102v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"atom-low-bit-quantization-for-efficient-and","repo_url":"https://github.com/efeslab/atom","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":null,"task_name":"GPU"},{"task_slug":"quantization","task_name":"Quantization"},{"task_slug":"sentiment-analysis","task_name":"Sentiment Analysis"}],"methods":[{"method_slug":null,"method_name":null},{"method_slug":"speed","method_name":"SPEED"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":null,"atlas_url":"https://app.syntology.ai/?focus=2310.19102","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2310.19102"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/efeslab/atom","reach":{"status":"ok"}}],"summary":{"ran":5,"unverified":5},"by_repo_kind":{"official":{"samples":10,"ran":5,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":10,"samples":[{"code_sha256_prefix":"6fa49d4516aa4cc9","entry":"bench","repo":"efeslab/atom","repo_kind":"official","path":"e2e/punica-atom/benchmarks/benchmark_utils.py","file_url":"https://github.com/efeslab/atom/blob/HEAD/e2e/punica-atom/benchmarks/benchmark_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"6fa49d4516aa4cc9"}},{"code_sha256_prefix":"fcfaa6b84e1d5ed8","entry":"generate_request_set","repo":"efeslab/atom","repo_kind":"official","path":"e2e/punica-atom/benchmarks/bench_textgen.py","file_url":"https://github.com/efeslab/atom/blob/HEAD/e2e/punica-atom/benchmarks/bench_textgen.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"fcfaa6b84e1d5ed8"}},{"code_sha256_prefix":"eb9d2957b3161128","entry":"get_c4","repo":"efeslab/atom","repo_kind":"official","path":"model/datautils.py","file_url":"https://github.com/efeslab/atom/blob/HEAD/model/datautils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"eb9d2957b3161128"}},{"code_sha256_prefix":"16d5da30ed40b3b7","entry":"rotary_pos_emb","repo":"efeslab/atom","repo_kind":"official","path":"e2e/punica-atom/punica/models/llama.py","file_url":"https://github.com/efeslab/atom/blob/HEAD/e2e/punica-atom/punica/models/llama.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"16d5da30ed40b3b7"}},{"code_sha256_prefix":"e582efeb977b565c","entry":"rotary_pos_emb","repo":"efeslab/atom","repo_kind":"official","path":"e2e/punica-atom/punica/models/llama_lora.py","file_url":"https://github.com/efeslab/atom/blob/HEAD/e2e/punica-atom/punica/models/llama_lora.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"e582efeb977b565c"}},{"code_sha256_prefix":"d71aa8d60119d553","entry":"get_ptb","repo":"efeslab/atom","repo_kind":"official","path":"model/datautils.py","file_url":"https://github.com/efeslab/atom/blob/HEAD/model/datautils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"d71aa8d60119d553"}},{"code_sha256_prefix":"f9f3ddd2d74b2929","entry":"get_wikitext2","repo":"efeslab/atom","repo_kind":"official","path":"model/datautils.py","file_url":"https://github.com/efeslab/atom/blob/HEAD/model/datautils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"f9f3ddd2d74b2929"}},{"code_sha256_prefix":"bbf10851b29f1daf","entry":"run_in_subprocess","repo":"efeslab/atom","repo_kind":"official","path":"e2e/punica-atom/benchmarks/bench_textgen_all.py","file_url":"https://github.com/efeslab/atom/blob/HEAD/e2e/punica-atom/benchmarks/bench_textgen_all.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"bbf10851b29f1daf"}},{"code_sha256_prefix":"f8a7dbaa50c5b09d","entry":"textgen_hf_pad","repo":"efeslab/atom","repo_kind":"official","path":"e2e/punica-atom/benchmarks/bench_textgen.py","file_url":"https://github.com/efeslab/atom/blob/HEAD/e2e/punica-atom/benchmarks/bench_textgen.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"f8a7dbaa50c5b09d"}},{"code_sha256_prefix":"db3a88d711087f21","entry":"textgen_punica","repo":"efeslab/atom","repo_kind":"official","path":"e2e/punica-atom/benchmarks/bench_textgen.py","file_url":"https://github.com/efeslab/atom/blob/HEAD/e2e/punica-atom/benchmarks/bench_textgen.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"db3a88d711087f21"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}