{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/towards-end-to-end-4-bit-inference-on","title":"QUIK: Towards End-to-End 4-Bit Inference on Generative Large Language Models","arxiv_id":"2310.09259","date":"2023-10-13","proceeding":null,"authors":["Saleh Ashkboos","Ilia Markov","Elias Frantar","Tingxuan Zhong","Xincheng Wang","Jie Ren","Torsten Hoefler","Dan Alistarh"],"abstract":"Large Language Models (LLMs) from the GPT family have become extremely popular, leading to a race towards reducing their inference costs to allow for efficient local computation. Yet, the vast majority of existing work focuses on weight-only quantization, which can reduce runtime costs in the memory-bound one-token-at-a-time generative setting, but does not address them in compute-bound scenarios, such as batched inference or prompt processing. In this paper, we address the general quantization problem, where both weights and activations should be quantized. We show, for the first time, that the majority of inference computations for large generative models such as LLaMA, OPT, and Falcon can be performed with both weights and activations being cast to 4 bits, in a way that leads to practical speedups, while at the same time maintaining good accuracy. We achieve this via a hybrid quantization strategy called QUIK, which compresses most of the weights and activations to 4-bit, while keeping some outlier weights and activations in higher-precision. The key feature of our scheme is that it is designed with computational efficiency in mind: we provide GPU kernels matching the QUIK format with highly-efficient layer-wise runtimes, which lead to practical end-to-end throughput improvements of up to 3.4x relative to FP16 execution. We provide detailed studies for models from the OPT, LLaMA-2 and Falcon families, as well as a first instance of accurate inference using quantization plus 2:4 sparsity. Code is available at: https://github.com/IST-DASLab/QUIK.","url_abs":"https://arxiv.org/abs/2310.09259v2","url_pdf":"https://arxiv.org/pdf/2310.09259v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"towards-end-to-end-4-bit-inference-on","repo_url":"https://github.com/ist-daslab/quik","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"computational-efficiency","task_name":"Computational Efficiency"},{"task_slug":null,"task_name":"GPU"},{"task_slug":"quantization","task_name":"Quantization"}],"methods":[{"method_slug":"adam","method_name":"Adam"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"attention-dropout","method_name":"Attention Dropout"},{"method_slug":"bpe","method_name":"BPE"},{"method_slug":"cosine-annealing","method_name":"Cosine Annealing"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"discriminative-fine-tuning","method_name":"Discriminative Fine-Tuning"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"gpt","method_name":"GPT"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"linear-warmup-with-cosine-annealing","method_name":"Linear Warmup With Cosine Annealing"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"opt","method_name":"OPT"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"weight-decay","method_name":"Weight Decay"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2310.09259","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2310.09259"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/ist-daslab/quik","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran":9,"unverified":4},"by_repo_kind":{"official":{"samples":13,"ran":9,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"aa2f6205ec9bf13c","entry":"asymmetric_quantize","repo":"ist-daslab/quik","repo_kind":"official","path":"experiments/quant_sim.py","file_url":"https://github.com/ist-daslab/quik/blob/HEAD/experiments/quant_sim.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"aa2f6205ec9bf13c"}},{"code_sha256_prefix":"980b7a8e42ae4a52","entry":"get_c4","repo":"ist-daslab/quik","repo_kind":"official","path":"experiments/datautils.py","file_url":"https://github.com/ist-daslab/quik/blob/HEAD/experiments/datautils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"980b7a8e42ae4a52"}},{"code_sha256_prefix":"6a7807d7d7ad0fb4","entry":"get_fp_features_num","repo":"ist-daslab/quik","repo_kind":"official","path":"experiments/modelutils.py","file_url":"https://github.com/ist-daslab/quik/blob/HEAD/experiments/modelutils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"6a7807d7d7ad0fb4"}},{"code_sha256_prefix":"f0ae4c8287c19664","entry":"get_fp_features_num","repo":"ist-daslab/quik","repo_kind":"official","path":"experiments/llama.py","file_url":"https://github.com/ist-daslab/quik/blob/HEAD/experiments/llama.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"f0ae4c8287c19664"}},{"code_sha256_prefix":"afa2292fc283853b","entry":"get_opt","repo":"ist-daslab/quik","repo_kind":"official","path":"experiments/modelutils.py","file_url":"https://github.com/ist-daslab/quik/blob/HEAD/experiments/modelutils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"afa2292fc283853b"}},{"code_sha256_prefix":"614c29352fee019e","entry":"get_ptb","repo":"ist-daslab/quik","repo_kind":"official","path":"experiments/datautils.py","file_url":"https://github.com/ist-daslab/quik/blob/HEAD/experiments/datautils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"614c29352fee019e"}},{"code_sha256_prefix":"715c101a77c1534c","entry":"pack_to_i4","repo":"ist-daslab/quik","repo_kind":"official","path":"experiments/qlinear.py","file_url":"https://github.com/ist-daslab/quik/blob/HEAD/experiments/qlinear.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"715c101a77c1534c"}},{"code_sha256_prefix":"417a0b6e6d05ce3b","entry":"symmetric_quantize","repo":"ist-daslab/quik","repo_kind":"official","path":"experiments/quant_sim.py","file_url":"https://github.com/ist-daslab/quik/blob/HEAD/experiments/quant_sim.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"417a0b6e6d05ce3b"}},{"code_sha256_prefix":"77a12785d85226d6","entry":"two_compl","repo":"ist-daslab/quik","repo_kind":"official","path":"experiments/qlinear.py","file_url":"https://github.com/ist-daslab/quik/blob/HEAD/experiments/qlinear.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"77a12785d85226d6"}},{"code_sha256_prefix":"586dfe618a4fadd4","entry":"get_falcon","repo":"ist-daslab/quik","repo_kind":"official","path":"experiments/fake_quant/modelutils.py","file_url":"https://github.com/ist-daslab/quik/blob/HEAD/experiments/fake_quant/modelutils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"586dfe618a4fadd4"}},{"code_sha256_prefix":"afe299403d03afcb","entry":"get_opt","repo":"ist-daslab/quik","repo_kind":"official","path":"experiments/fake_quant/modelutils.py","file_url":"https://github.com/ist-daslab/quik/blob/HEAD/experiments/fake_quant/modelutils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"afe299403d03afcb"}},{"code_sha256_prefix":"79092c10d8a25941","entry":"get_wikitext2","repo":"ist-daslab/quik","repo_kind":"official","path":"experiments/datautils.py","file_url":"https://github.com/ist-daslab/quik/blob/HEAD/experiments/datautils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"79092c10d8a25941"}},{"code_sha256_prefix":"340f10d2f7201ee7","entry":"llama_eval","repo":"ist-daslab/quik","repo_kind":"official","path":"experiments/llama.py","file_url":"https://github.com/ist-daslab/quik/blob/HEAD/experiments/llama.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"340f10d2f7201ee7"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}