{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/vitality-unifying-low-rank-and-sparse","title":"ViTALiTy: Unifying Low-rank and Sparse Approximation for Vision Transformer Acceleration with a Linear Taylor Attention","arxiv_id":"2211.05109","date":"2022-11-09","proceeding":null,"authors":["Jyotikrishna Dass","Shang Wu","Huihong Shi","Chaojian Li","Zhifan Ye","Zhongfeng Wang","Yingyan Lin"],"abstract":"Vision Transformer (ViT) has emerged as a competitive alternative to convolutional neural networks for various computer vision applications. Specifically, ViT multi-head attention layers make it possible to embed information globally across the overall image. Nevertheless, computing and storing such attention matrices incurs a quadratic cost dependency on the number of patches, limiting its achievable efficiency and scalability and prohibiting more extensive real-world ViT applications on resource-constrained devices. Sparse attention has been shown to be a promising direction for improving hardware acceleration efficiency for NLP models. However, a systematic counterpart approach is still missing for accelerating ViT models. To close the above gap, we propose a first-of-its-kind algorithm-hardware codesigned framework, dubbed ViTALiTy, for boosting the inference efficiency of ViTs. Unlike sparsity-based Transformer accelerators for NLP, ViTALiTy unifies both low-rank and sparse components of the attention in ViTs. At the algorithm level, we approximate the dot-product softmax operation via first-order Taylor attention with row-mean centering as the low-rank component to linearize the cost of attention blocks and further boost the accuracy by incorporating a sparsity-based regularization. At the hardware level, we develop a dedicated accelerator to better leverage the resulting workload and pipeline from ViTALiTy's linear Taylor attention which requires the execution of only the low-rank component, to further boost the hardware efficiency. Extensive experiments and ablation studies validate that ViTALiTy offers boosted end-to-end efficiency (e.g., $3\\times$ faster and $3\\times$ energy-efficient) under comparable accuracy, with respect to the state-of-the-art solution.","url_abs":"https://arxiv.org/abs/2211.05109v1","url_pdf":"https://arxiv.org/pdf/2211.05109v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"vitality-unifying-low-rank-and-sparse","repo_url":"https://github.com/GATECH-EIC/ViTaLiTy","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[],"methods":[{"method_slug":"absolute-position-encodings","method_name":"Absolute Position Encodings"},{"method_slug":"adam","method_name":"Adam"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"bpe","method_name":"BPE"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"label-smoothing","method_name":"Label Smoothing"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"position-wise-feed-forward-layer","method_name":"Position-Wise Feed-Forward Layer"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"transformer","method_name":"Transformer"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2211.05109","atlas_url":"https://app.syntology.ai/?focus=2211.05109","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2211.05109"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/GATECH-EIC/ViTaLiTy","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran":9,"ran_draft_wrong":1,"ran_fixture":1,"ran_honours":1,"unverified":2},"by_repo_kind":{"official":{"samples":14,"ran":12,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":14,"samples":[{"code_sha256_prefix":"b008c1f4638ba65f","entry":"calc_max_quant_value","repo":"GATECH-EIC/ViTaLiTy","repo_kind":"official","path":"src/quant_utils.py","file_url":"https://github.com/GATECH-EIC/ViTaLiTy/blob/HEAD/src/quant_utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":false,"mcp_get_code":{"code_sha256":"b008c1f4638ba65f"}},{"code_sha256_prefix":"750c588ade0c659c","entry":"calculate_qparams","repo":"GATECH-EIC/ViTaLiTy","repo_kind":"official","path":"src/quantize.py","file_url":"https://github.com/GATECH-EIC/ViTaLiTy/blob/HEAD/src/quantize.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":false,"mcp_get_code":{"code_sha256":"750c588ade0c659c"}},{"code_sha256_prefix":"e90596193a6f191a","entry":"conv3x3","repo":"GATECH-EIC/ViTaLiTy","repo_kind":"official","path":"src/patchconvnet_models.py","file_url":"https://github.com/GATECH-EIC/ViTaLiTy/blob/HEAD/src/patchconvnet_models.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":false,"mcp_get_code":{"code_sha256":"e90596193a6f191a"}},{"code_sha256_prefix":"33efc9f1ccc7933c","entry":"drop_block_2d","repo":"GATECH-EIC/ViTaLiTy","repo_kind":"official","path":"src/drop.py","file_url":"https://github.com/GATECH-EIC/ViTaLiTy/blob/HEAD/src/drop.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":false,"mcp_get_code":{"code_sha256":"33efc9f1ccc7933c"}},{"code_sha256_prefix":"30d63ccefb97a166","entry":"drop_block_fast_2d","repo":"GATECH-EIC/ViTaLiTy","repo_kind":"official","path":"src/drop.py","file_url":"https://github.com/GATECH-EIC/ViTaLiTy/blob/HEAD/src/drop.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":false,"mcp_get_code":{"code_sha256":"30d63ccefb97a166"}},{"code_sha256_prefix":"3ac6b7d76e8e3584","entry":"drop_path","repo":"GATECH-EIC/ViTaLiTy","repo_kind":"official","path":"src/drop.py","file_url":"https://github.com/GATECH-EIC/ViTaLiTy/blob/HEAD/src/drop.py","link_basis":"harvester_set","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":false,"mcp_get_code":{"code_sha256":"3ac6b7d76e8e3584"}},{"code_sha256_prefix":"4768845394611c2a","entry":"get_activation","repo":"GATECH-EIC/ViTaLiTy","repo_kind":"official","path":"src/mlp.py","file_url":"https://github.com/GATECH-EIC/ViTaLiTy/blob/HEAD/src/mlp.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":false,"mcp_get_code":{"code_sha256":"4768845394611c2a"}},{"code_sha256_prefix":"b2f5a5d7be746e1f","entry":"get_dynamic_scale","repo":"GATECH-EIC/ViTaLiTy","repo_kind":"official","path":"src/quant_utils.py","file_url":"https://github.com/GATECH-EIC/ViTaLiTy/blob/HEAD/src/quant_utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":false,"mcp_get_code":{"code_sha256":"b2f5a5d7be746e1f"}},{"code_sha256_prefix":"66ea78e93ec5b48e","entry":"get_grad","repo":"GATECH-EIC/ViTaLiTy","repo_kind":"official","path":"src/quantize.py","file_url":"https://github.com/GATECH-EIC/ViTaLiTy/blob/HEAD/src/quantize.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":false,"mcp_get_code":{"code_sha256":"66ea78e93ec5b48e"}},{"code_sha256_prefix":"1184aa6c5f001630","entry":"get_scale","repo":"GATECH-EIC/ViTaLiTy","repo_kind":"official","path":"src/quant_utils.py","file_url":"https://github.com/GATECH-EIC/ViTaLiTy/blob/HEAD/src/quant_utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":false,"mcp_get_code":{"code_sha256":"1184aa6c5f001630"}},{"code_sha256_prefix":"24400bc088ed228d","entry":"make_divisible","repo":"GATECH-EIC/ViTaLiTy","repo_kind":"official","path":"src/patch_embed.py","file_url":"https://github.com/GATECH-EIC/ViTaLiTy/blob/HEAD/src/patch_embed.py","link_basis":"harvester_set","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":false,"mcp_get_code":{"code_sha256":"24400bc088ed228d"}},{"code_sha256_prefix":"3f1de05d36b1a783","entry":"quantize","repo":"GATECH-EIC/ViTaLiTy","repo_kind":"official","path":"src/quantize.py","file_url":"https://github.com/GATECH-EIC/ViTaLiTy/blob/HEAD/src/quantize.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":false,"mcp_get_code":{"code_sha256":"3f1de05d36b1a783"}},{"code_sha256_prefix":"9153241c4331eb1a","entry":"build_dataset","repo":"GATECH-EIC/ViTaLiTy","repo_kind":"official","path":"src/datasets.py","file_url":"https://github.com/GATECH-EIC/ViTaLiTy/blob/HEAD/src/datasets.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":false,"mcp_get_code":{"code_sha256":"9153241c4331eb1a"}},{"code_sha256_prefix":"3bc74137ab36aa79","entry":"build_transform","repo":"GATECH-EIC/ViTaLiTy","repo_kind":"official","path":"src/datasets.py","file_url":"https://github.com/GATECH-EIC/ViTaLiTy/blob/HEAD/src/datasets.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":false,"mcp_get_code":{"code_sha256":"3bc74137ab36aa79"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}