{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/fq-vit-fully-quantized-vision-transformer","title":"FQ-ViT: Post-Training Quantization for Fully Quantized Vision Transformer","arxiv_id":"2111.13824","date":"2021-11-27","proceeding":null,"authors":["Yang Lin","Tianyu Zhang","Peiqin Sun","Zheng Li","Shuchang Zhou"],"abstract":"Network quantization significantly reduces model inference complexity and has been widely used in real-world deployments. However, most existing quantization methods have been developed mainly on Convolutional Neural Networks (CNNs), and suffer severe degradation when applied to fully quantized vision transformers. In this work, we demonstrate that many of these difficulties arise because of serious inter-channel variation in LayerNorm inputs, and present, Power-of-Two Factor (PTF), a systematic method to reduce the performance degradation and inference complexity of fully quantized vision transformers. In addition, observing an extreme non-uniform distribution in attention maps, we propose Log-Int-Softmax (LIS) to sustain that and simplify inference by using 4-bit quantization and the BitShift operator. Comprehensive experiments on various transformer-based architectures and benchmarks show that our Fully Quantized Vision Transformer (FQ-ViT) outperforms previous works while even using lower bit-width on attention maps. For instance, we reach 84.89% top-1 accuracy with ViT-L on ImageNet and 50.8 mAP with Cascade Mask R-CNN (Swin-S) on COCO. To our knowledge, we are the first to achieve lossless accuracy degradation (~1%) on fully quantized vision transformers. The code is available at https://github.com/megvii-research/FQ-ViT.","url_abs":"https://arxiv.org/abs/2111.13824v4","url_pdf":"https://arxiv.org/pdf/2111.13824v4.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"fq-vit-fully-quantized-vision-transformer","repo_url":"https://github.com/megvii-research/FQ-ViT","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"quantization","task_name":"Quantization"}],"methods":[{"method_slug":"absolute-position-encodings","method_name":"Absolute Position Encodings"},{"method_slug":"adam","method_name":"Adam"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"bpe","method_name":"BPE"},{"method_slug":"cascade-mask-r-cnn","method_name":"Cascade Mask R-CNN"},{"method_slug":"convolution","method_name":"Convolution"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"label-smoothing","method_name":"Label Smoothing"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"mask-r-cnn","method_name":"Mask R-CNN"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"position-wise-feed-forward-layer","method_name":"Position-Wise Feed-Forward Layer"},{"method_slug":"rpn","method_name":"RPN"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"roi-align","method_name":"RoIAlign"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"transformer","method_name":"Transformer"},{"method_slug":"vision-transformer","method_name":"Vision Transformer"}],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/quantization-on-imagenet","task":"Quantization","dataset":"ImageNet","model":"FQ-ViT (ViT-L)","rank_in_archive_order":1,"of":27,"metrics":{"Activation bits":"8","Top-1 Accuracy (%)":"85.03","Weight bits":"8"},"uses_additional_data":false},{"leaderboard":"/sota/quantization-on-imagenet","task":"Quantization","dataset":"ImageNet","model":"FQ-ViT (ViT-B)","rank_in_archive_order":2,"of":27,"metrics":{"Activation bits":"8","Top-1 Accuracy (%)":"83.31","Weight bits":"8"},"uses_additional_data":false},{"leaderboard":"/sota/quantization-on-imagenet","task":"Quantization","dataset":"ImageNet","model":"FQ-ViT (Swin-B)","rank_in_archive_order":3,"of":27,"metrics":{"Activation bits":"8","Top-1 Accuracy (%)":"82.97","Weight bits":"8"},"uses_additional_data":false},{"leaderboard":"/sota/quantization-on-imagenet","task":"Quantization","dataset":"ImageNet","model":"FQ-ViT (Swin-S)","rank_in_archive_order":4,"of":27,"metrics":{"Activation bits":"8","Top-1 Accuracy (%)":"82.71","Weight bits":"8"},"uses_additional_data":false},{"leaderboard":"/sota/quantization-on-imagenet","task":"Quantization","dataset":"ImageNet","model":"FQ-ViT (DeiT-B)","rank_in_archive_order":5,"of":27,"metrics":{"Activation bits":"8","Top-1 Accuracy (%)":"81.20","Weight bits":"8"},"uses_additional_data":false},{"leaderboard":"/sota/quantization-on-imagenet","task":"Quantization","dataset":"ImageNet","model":"FQ-ViT (Swin-T)","rank_in_archive_order":6,"of":27,"metrics":{"Activation bits":"8","Top-1 Accuracy (%)":"80.51","Weight bits":"8"},"uses_additional_data":false},{"leaderboard":"/sota/quantization-on-imagenet","task":"Quantization","dataset":"ImageNet","model":"FQ-ViT (DeiT-S)","rank_in_archive_order":7,"of":27,"metrics":{"Activation bits":"8","Top-1 Accuracy (%)":"79.17","Weight bits":"8"},"uses_additional_data":false},{"leaderboard":"/sota/quantization-on-imagenet","task":"Quantization","dataset":"ImageNet","model":"FQ-ViT (DeiT-T)","rank_in_archive_order":21,"of":27,"metrics":{"Activation bits":"8","Top-1 Accuracy (%)":"71.61","Weight bits":"8"},"uses_additional_data":false}],"syntology":{"syntology_url":"https://syntology.ai/paper/2111.13824","atlas_url":"https://app.syntology.ai/?focus=2111.13824","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2111.13824"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/megvii-research/FQ-ViT","reach":null}],"summary":{"ran_fixture":1,"ran_draft_wrong":1,"ran_honours":1},"by_repo_kind":{"official":{"samples":2,"ran":2,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":1,"samples":[{"code_sha256_prefix":"18badd533b52adb3","entry":"accuracy","repo":null,"repo_kind":null,"path":null,"file_url":null,"link_basis":"identical_code_first_harvested_elsewhere","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":null,"inline_ok":false,"mcp_get_code":{"code_sha256":"18badd533b52adb3"}},{"code_sha256_prefix":"e4b6f33d7173cf96","entry":"str2model","repo":"megvii-research/FQ-ViT","repo_kind":"official","path":"test_quant.py","file_url":"https://github.com/megvii-research/FQ-ViT/blob/HEAD/test_quant.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"e4b6f33d7173cf96"}},{"code_sha256_prefix":"0e215bc8d251972f","entry":"validate","repo":"megvii-research/FQ-ViT","repo_kind":"official","path":"test_quant.py","file_url":"https://github.com/megvii-research/FQ-ViT/blob/HEAD/test_quant.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"0e215bc8d251972f"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}