{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/gptq-accurate-post-training-quantization-for","title":"GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers","arxiv_id":"2210.17323","date":"2022-10-31","proceeding":null,"authors":["Elias Frantar","Saleh Ashkboos","Torsten Hoefler","Dan Alistarh"],"abstract":"Generative Pre-trained Transformer models, known as GPT or OPT, set themselves apart through breakthrough performance across complex language modelling tasks, but also by their extremely high computational and storage costs. Specifically, due to their massive size, even inference for large, highly-accurate GPT models may require multiple performant GPUs, which limits the usability of such models. While there is emerging work on relieving this pressure via model compression, the applicability and performance of existing compression techniques is limited by the scale and complexity of GPT models. In this paper, we address this challenge, and propose GPTQ, a new one-shot weight quantization method based on approximate second-order information, that is both highly-accurate and highly-efficient. Specifically, GPTQ can quantize GPT models with 175 billion parameters in approximately four GPU hours, reducing the bitwidth down to 3 or 4 bits per weight, with negligible accuracy degradation relative to the uncompressed baseline. Our method more than doubles the compression gains relative to previously-proposed one-shot quantization methods, preserving accuracy, allowing us for the first time to execute an 175 billion-parameter model inside a single GPU for generative inference. Moreover, we also show that our method can still provide reasonable accuracy in the extreme quantization regime, in which weights are quantized to 2-bit or even ternary quantization levels. We show experimentally that these improvements can be leveraged for end-to-end inference speedups over FP16, of around 3.25x when using high-end GPUs (NVIDIA A100) and 4.5x when using more cost-effective ones (NVIDIA A6000). The implementation is available at https://github.com/IST-DASLab/gptq.","url_abs":"https://arxiv.org/abs/2210.17323v2","url_pdf":"https://arxiv.org/pdf/2210.17323v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"gptq-accurate-post-training-quantization-for","repo_url":"https://github.com/ist-daslab/gptq","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"gptq-accurate-post-training-quantization-for","repo_url":"https://github.com/bentoml/openllm","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok","spdx":"Apache-2.0"}},{"paper_slug":"gptq-accurate-post-training-quantization-for","repo_url":"https://github.com/cornell-zhang/llm-datatypes","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"tf","reach":null},{"paper_slug":"gptq-accurate-post-training-quantization-for","repo_url":"https://github.com/dfki-nlp/llmcheckup","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}},{"paper_slug":"gptq-accurate-post-training-quantization-for","repo_url":"https://github.com/ecolab-postech/owq","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"gptq-accurate-post-training-quantization-for","repo_url":"https://github.com/eth-easl/deltazip","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"gptq-accurate-post-training-quantization-for","repo_url":"https://github.com/greenbitai/low_bit_llama","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"gptq-accurate-post-training-quantization-for","repo_url":"https://github.com/huggingface/text-generation-inference","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"gptq-accurate-post-training-quantization-for","repo_url":"https://github.com/microsoft/bitblas","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"gptq-accurate-post-training-quantization-for","repo_url":"https://github.com/mit-han-lab/llm-awq","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"gptq-accurate-post-training-quantization-for","repo_url":"https://github.com/qwopqwop200/GPTQ-for-LLaMa","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"gptq-accurate-post-training-quantization-for","repo_url":"https://github.com/rasmith/vllm","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"jax","reach":{"status":"ok","spdx":"Apache-2.0"}},{"paper_slug":"gptq-accurate-post-training-quantization-for","repo_url":"https://github.com/thudm/chatglm-6b","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}},{"paper_slug":"gptq-accurate-post-training-quantization-for","repo_url":"https://github.com/vectorch-ai/scalellm","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"gptq-accurate-post-training-quantization-for","repo_url":"https://github.com/vllm-project/vllm","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"gptq-accurate-post-training-quantization-for","repo_url":"https://github.com/xvyaward/owq","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"gptq-accurate-post-training-quantization-for","repo_url":"https://github.com/PanQiWei/AutoGPTQ","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":null,"task_name":"GPU"},{"task_slug":"language-modelling","task_name":"Language Modelling"},{"task_slug":"model-compression","task_name":"Model Compression"},{"task_slug":"quantization","task_name":"Quantization"}],"methods":[{"method_slug":"absolute-position-encodings","method_name":"Absolute Position Encodings"},{"method_slug":"adam","method_name":"Adam"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"attention-dropout","method_name":"Attention Dropout"},{"method_slug":"bpe","method_name":"BPE"},{"method_slug":"cosine-annealing","method_name":"Cosine Annealing"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"discriminative-fine-tuning","method_name":"Discriminative Fine-Tuning"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"gpt","method_name":"GPT"},{"method_slug":"label-smoothing","method_name":"Label Smoothing"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"linear-warmup-with-cosine-annealing","method_name":"Linear Warmup With Cosine Annealing"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"opt","method_name":"OPT"},{"method_slug":"position-wise-feed-forward-layer","method_name":"Position-Wise Feed-Forward Layer"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"transformer","method_name":"Transformer"},{"method_slug":"weight-decay","method_name":"Weight Decay"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2210.17323","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2210.17323"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/IST-DASLab/gptq","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/cornell-zhang/llm-datatypes","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/vllm-project/vllm","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/xvyaward/owq","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/mit-han-lab/llm-awq","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/bentoml/openllm","reach":{"status":"ok","spdx":"Apache-2.0"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/dfki-nlp/llmcheckup","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/vectorch-ai/scalellm","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/PanQiWei/AutoGPTQ","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/eth-easl/deltazip","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/rasmith/vllm","reach":{"status":"ok","spdx":"Apache-2.0"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/qwopqwop200/GPTQ-for-LLaMa","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/ecolab-postech/owq","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/thudm/chatglm-6b","reach":{"status":"ok","spdx":"Apache-2.0"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/greenbitai/low_bit_llama","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/ist-daslab/gptq","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/huggingface/text-generation-inference","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/microsoft/bitblas","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":2,"ran_draft_wrong":3,"unverified":10},"by_repo_kind":{"official":{"samples":4,"ran":1,"repositories":1},"listed":{"samples":11,"ran":4,"repositories":6}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":1,"samples":[{"code_sha256_prefix":"6729f9f2a233d315","entry":"GPTQ","repo":"PanQiWei/AutoGPTQ","repo_kind":"listed","path":"auto_gptq/quantization/gptq.py","file_url":"https://github.com/PanQiWei/AutoGPTQ/blob/HEAD/auto_gptq/quantization/gptq.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"6729f9f2a233d315"}},{"code_sha256_prefix":"c1a5aa86dbaf659b","entry":"Logger","repo":"cornell-zhang/llm-datatypes","repo_kind":"listed","path":"neural_compressor/torch/algorithms/weight_only/gptq.py","file_url":"https://github.com/cornell-zhang/llm-datatypes/blob/HEAD/neural_compressor/torch/algorithms/weight_only/gptq.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"c1a5aa86dbaf659b"}},{"code_sha256_prefix":"887f43fe08a6472d","entry":"get_opt","repo":"IST-DASLab/gptq","repo_kind":"official","path":"opt.py","file_url":"https://github.com/IST-DASLab/gptq/blob/HEAD/opt.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"887f43fe08a6472d"}},{"code_sha256_prefix":"a855208c73825af0","entry":"torch_snr_error","repo":"qwopqwop200/GPTQ-for-LLaMa","repo_kind":"listed","path":"gptq.py","file_url":"https://github.com/qwopqwop200/GPTQ-for-LLaMa/blob/HEAD/gptq.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"a855208c73825af0"}},{"code_sha256_prefix":"671302aaa20f0ee9","entry":"torch_snr_error","repo":"huggingface/text-generation-inference","repo_kind":"listed","path":"server/text_generation_server/layers/gptq/quantize.py","file_url":"https://github.com/huggingface/text-generation-inference/blob/HEAD/server/text_generation_server/layers/gptq/quantize.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"671302aaa20f0ee9"}},{"code_sha256_prefix":"6164b2bb82400dba","entry":"GPTQ","repo":"qwopqwop200/GPTQ-for-LLaMa","repo_kind":"listed","path":"gptq.py","file_url":"https://github.com/qwopqwop200/GPTQ-for-LLaMa/blob/HEAD/gptq.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"6164b2bb82400dba"}},{"code_sha256_prefix":"f4d20c4a7bccc9a2","entry":"GPTQ","repo":"cornell-zhang/llm-datatypes","repo_kind":"listed","path":"neural_compressor/torch/algorithms/weight_only/gptq.py","file_url":"https://github.com/cornell-zhang/llm-datatypes/blob/HEAD/neural_compressor/torch/algorithms/weight_only/gptq.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"f4d20c4a7bccc9a2"}},{"code_sha256_prefix":"cdbcb9ec70b30c58","entry":"GPTQ","repo":"ist-daslab/gptq","repo_kind":"official","path":"gptq.py","file_url":"https://github.com/ist-daslab/gptq/blob/HEAD/gptq.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"cdbcb9ec70b30c58"}},{"code_sha256_prefix":"0747a1d535734929","entry":"GPTQ","repo":"huggingface/text-generation-inference","repo_kind":"listed","path":"server/text_generation_server/layers/gptq/quantize.py","file_url":"https://github.com/huggingface/text-generation-inference/blob/HEAD/server/text_generation_server/layers/gptq/quantize.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"0747a1d535734929"}},{"code_sha256_prefix":"c30c4d1c0c0838ff","entry":"GPTQ_OWQ","repo":"ecolab-postech/owq","repo_kind":"listed","path":"owq/recon.py","file_url":"https://github.com/ecolab-postech/owq/blob/HEAD/owq/recon.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"c30c4d1c0c0838ff"}},{"code_sha256_prefix":"2e160e59dad498ff","entry":"Kernel","repo":"vectorch-ai/scalellm","repo_kind":"listed","path":"src/kernels/quantization/marlin/generate_instantiations.py","file_url":"https://github.com/vectorch-ai/scalellm/blob/HEAD/src/kernels/quantization/marlin/generate_instantiations.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"2e160e59dad498ff"}},{"code_sha256_prefix":"dd6638bb4690fc4c","entry":"Quantizer","repo":"cornell-zhang/llm-datatypes","repo_kind":"listed","path":"neural_compressor/torch/algorithms/weight_only/gptq.py","file_url":"https://github.com/cornell-zhang/llm-datatypes/blob/HEAD/neural_compressor/torch/algorithms/weight_only/gptq.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"dd6638bb4690fc4c"}},{"code_sha256_prefix":"7452671bd2402cf2","entry":"Quantizer","repo":"huggingface/text-generation-inference","repo_kind":"listed","path":"server/text_generation_server/layers/gptq/quantize.py","file_url":"https://github.com/huggingface/text-generation-inference/blob/HEAD/server/text_generation_server/layers/gptq/quantize.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"7452671bd2402cf2"}},{"code_sha256_prefix":"5a1ab8a11275fe3d","entry":"opt_pack3","repo":"IST-DASLab/gptq","repo_kind":"official","path":"opt.py","file_url":"https://github.com/IST-DASLab/gptq/blob/HEAD/opt.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"5a1ab8a11275fe3d"}},{"code_sha256_prefix":"60602426146a6763","entry":"opt_sequential","repo":"IST-DASLab/gptq","repo_kind":"official","path":"opt.py","file_url":"https://github.com/IST-DASLab/gptq/blob/HEAD/opt.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"60602426146a6763"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}