{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/vit-1-58b-mobile-vision-transformers-in-the-1","title":"ViT-1.58b: Mobile Vision Transformers in the 1-bit Era","arxiv_id":"2406.18051","date":"2024-06-26","proceeding":null,"authors":["Zhengqing Yuan","Rong Zhou","Hongyi Wang","Lifang He","Yanfang Ye","Lichao Sun"],"abstract":"Vision Transformers (ViTs) have achieved remarkable performance in various image classification tasks by leveraging the attention mechanism to process image patches as tokens. However, the high computational and memory demands of ViTs pose significant challenges for deployment in resource-constrained environments. This paper introduces ViT-1.58b, a novel 1.58-bit quantized ViT model designed to drastically reduce memory and computational overhead while preserving competitive performance. ViT-1.58b employs ternary quantization, which refines the balance between efficiency and accuracy by constraining weights to {-1, 0, 1} and quantizing activations to 8-bit precision. Our approach ensures efficient scaling in terms of both memory and computation. Experiments on CIFAR-10 and ImageNet-1k demonstrate that ViT-1.58b maintains comparable accuracy to full-precision Vit, with significant reductions in memory usage and computational costs. This paper highlights the potential of extreme quantization techniques in developing sustainable AI solutions and contributes to the broader discourse on efficient model deployment in practical applications. Our code and weights are available at https://github.com/DLYuanGod/ViT-1.58b.","url_abs":"https://arxiv.org/abs/2406.18051v1","url_pdf":"https://arxiv.org/pdf/2406.18051v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"vit-1-58b-mobile-vision-transformers-in-the-1","repo_url":"https://github.com/dlyuangod/vit-1.58b","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":"image-classification","task_name":"Image Classification"},{"task_slug":"quantization","task_name":"Quantization"},{"task_slug":"image-classification","task_name":"image-classification"}],"methods":[{"method_slug":"attention","method_name":"Attention"},{"method_slug":"softmax","method_name":"Softmax"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2406.18051","atlas_url":"https://app.syntology.ai/?focus=2406.18051","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2406.18051"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/dlyuangod/vit-1.58b","reach":{"status":"ok"}}],"summary":{"ran":5,"ran_draft_wrong":2,"ran_violates":1,"ran_fixture":1},"by_repo_kind":{"official":{"samples":9,"ran":9,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":9,"samples":[{"code_sha256_prefix":"9e106282d2e11561","entry":"activation_quant","repo":"dlyuangod/vit-1.58b","repo_kind":"official","path":"bitlinear.py","file_url":"https://github.com/dlyuangod/vit-1.58b/blob/HEAD/bitlinear.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"9e106282d2e11561"}},{"code_sha256_prefix":"c16aee9490eb8729","entry":"eval_decorator","repo":"dlyuangod/vit-1.58b","repo_kind":"official","path":"BitNet/bitnet/at.py","file_url":"https://github.com/dlyuangod/vit-1.58b/blob/HEAD/BitNet/bitnet/at.py","link_basis":"plan_row","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"c16aee9490eb8729"}},{"code_sha256_prefix":"aa5486a3650902d8","entry":"exists","repo":"dlyuangod/vit-1.58b","repo_kind":"official","path":"BitNet/bitnet/at.py","file_url":"https://github.com/dlyuangod/vit-1.58b/blob/HEAD/BitNet/bitnet/at.py","link_basis":"harvester_set","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"aa5486a3650902d8"}},{"code_sha256_prefix":"1a1ee1200dc36a2f","entry":"get_init_weights_vit","repo":"dlyuangod/vit-1.58b","repo_kind":"official","path":"v.py","file_url":"https://github.com/dlyuangod/vit-1.58b/blob/HEAD/v.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"1a1ee1200dc36a2f"}},{"code_sha256_prefix":"18ae3e6ef2cf02ec","entry":"has_file_allowed_extension","repo":"dlyuangod/vit-1.58b","repo_kind":"official","path":"dataset_floder.py","file_url":"https://github.com/dlyuangod/vit-1.58b/blob/HEAD/dataset_floder.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"18ae3e6ef2cf02ec"}},{"code_sha256_prefix":"8ac50ae8773b9431","entry":"is_image_file","repo":"dlyuangod/vit-1.58b","repo_kind":"official","path":"dataset_floder.py","file_url":"https://github.com/dlyuangod/vit-1.58b/blob/HEAD/dataset_floder.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"8ac50ae8773b9431"}},{"code_sha256_prefix":"ad6afdce7eb73101","entry":"make_dataset","repo":"dlyuangod/vit-1.58b","repo_kind":"official","path":"dataset_floder.py","file_url":"https://github.com/dlyuangod/vit-1.58b/blob/HEAD/dataset_floder.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"ad6afdce7eb73101"}},{"code_sha256_prefix":"900ed88360f3d92f","entry":"resize_pos_embed","repo":"dlyuangod/vit-1.58b","repo_kind":"official","path":"v.py","file_url":"https://github.com/dlyuangod/vit-1.58b/blob/HEAD/v.py","link_basis":"plan_row","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"900ed88360f3d92f"}},{"code_sha256_prefix":"3b65d5d77510c964","entry":"top_k","repo":"dlyuangod/vit-1.58b","repo_kind":"official","path":"BitNet/bitnet/at.py","file_url":"https://github.com/dlyuangod/vit-1.58b/blob/HEAD/BitNet/bitnet/at.py","link_basis":"plan_row","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"3b65d5d77510c964"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}