{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/fast-vision-transformers-with-hilo-attention","title":"Fast Vision Transformers with HiLo Attention","arxiv_id":"2205.13213","date":"2022-05-26","proceeding":null,"authors":["Zizheng Pan","Jianfei Cai","Bohan Zhuang"],"abstract":"Vision Transformers (ViTs) have triggered the most recent and significant breakthroughs in computer vision. Their efficient designs are mostly guided by the indirect metric of computational complexity, i.e., FLOPs, which however has a clear gap with the direct metric such as throughput. Thus, we propose to use the direct speed evaluation on the target platform as the design principle for efficient ViTs. Particularly, we introduce LITv2, a simple and effective ViT which performs favourably against the existing state-of-the-art methods across a spectrum of different model sizes with faster speed. At the core of LITv2 is a novel self-attention mechanism, which we dub HiLo. HiLo is inspired by the insight that high frequencies in an image capture local fine details and low frequencies focus on global structures, whereas a multi-head self-attention layer neglects the characteristic of different frequencies. Therefore, we propose to disentangle the high/low frequency patterns in an attention layer by separating the heads into two groups, where one group encodes high frequencies via self-attention within each local window, and another group encodes low frequencies by performing global attention between the average-pooled low-frequency keys and values from each window and each query position in the input feature map. Benefiting from the efficient design for both groups, we show that HiLo is superior to the existing attention mechanisms by comprehensively benchmarking FLOPs, speed and memory consumption on GPUs and CPUs. For example, HiLo is 1.4x faster than spatial reduction attention and 1.6x faster than local window attention on CPUs. Powered by HiLo, LITv2 serves as a strong backbone for mainstream vision tasks including image classification, dense detection and segmentation. Code is available at https://github.com/ziplab/LITv2.","url_abs":"https://arxiv.org/abs/2205.13213v5","url_pdf":"https://arxiv.org/pdf/2205.13213v5.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"fast-vision-transformers-with-hilo-attention","repo_url":"https://github.com/zip-group/litv2","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}},{"paper_slug":"fast-vision-transformers-with-hilo-attention","repo_url":"https://github.com/ziplab/litv2","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}},{"paper_slug":"fast-vision-transformers-with-hilo-attention","repo_url":"https://github.com/MonashAI/LIT","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"fast-vision-transformers-with-hilo-attention","repo_url":"https://github.com/Westlake-AI/openmixup","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"fast-vision-transformers-with-hilo-attention","repo_url":"https://github.com/zhuang-group/lit","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"benchmarking","task_name":"Benchmarking"},{"task_slug":"efficient-vits","task_name":"Efficient ViTs"},{"task_slug":"image-classification","task_name":"Image Classification"},{"task_slug":"object-detection","task_name":"Object Detection"},{"task_slug":"semantic-segmentation","task_name":"Semantic Segmentation"},{"task_slug":"image-classification","task_name":"image-classification"}],"methods":[{"method_slug":"speed","method_name":"SPEED"}],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/image-classification-on-imagenet","task":"Image Classification","dataset":"ImageNet","model":"LITv2-B|384","rank_in_archive_order":300,"of":1060,"metrics":{"GFLOPs":"39.7","Number of params":"87M","Top 1 Accuracy":"84.7%"},"uses_additional_data":false},{"leaderboard":"/sota/image-classification-on-imagenet","task":"Image Classification","dataset":"ImageNet","model":"LITv2-B","rank_in_archive_order":406,"of":1060,"metrics":{"GFLOPs":"13.2","Top 1 Accuracy":"83.6%"},"uses_additional_data":false},{"leaderboard":"/sota/image-classification-on-imagenet","task":"Image Classification","dataset":"ImageNet","model":"LITv2-M","rank_in_archive_order":444,"of":1060,"metrics":{"GFLOPs":"7.5","Number of params":"49M","Top 1 Accuracy":"83.3%"},"uses_additional_data":false},{"leaderboard":"/sota/image-classification-on-imagenet","task":"Image Classification","dataset":"ImageNet","model":"LITv2-S","rank_in_archive_order":587,"of":1060,"metrics":{"GFLOPs":"3.7","Number of params":"28M","Top 1 Accuracy":"82%"},"uses_additional_data":false}],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2205.13213","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2205.13213"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/Westlake-AI/openmixup","reach":null},{"provenance":"deterministic:regex_extraction","url":"https://github.com/ziplab/LITv2","reach":{"status":"ok","spdx":"Apache-2.0"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/zip-group/litv2","reach":{"status":"ok","spdx":"Apache-2.0"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/ziplab/litv2","reach":{"status":"ok","spdx":"Apache-2.0"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/zhuang-group/lit","reach":{"status":"ok","spdx":"Apache-2.0"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/MonashAI/LIT","reach":null}],"summary":{"ran":1,"ran_draft_wrong":1,"unverified":11},"by_repo_kind":{"official":{"samples":1,"ran":1,"repositories":1},"listed":{"samples":12,"ran":1,"repositories":2}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"a4a70588d1c2a75e","entry":"RelPosAttention","repo":"MonashAI/LIT","repo_kind":"listed","path":"classification/code_for_lit_s_m_b/models/attentions.py","file_url":"https://github.com/MonashAI/LIT/blob/HEAD/classification/code_for_lit_s_m_b/models/attentions.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"a4a70588d1c2a75e"}},{"code_sha256_prefix":"d9590a53c9850a4d","entry":"flops_to_string","repo":"ziplab/LITv2","repo_kind":"official","path":"classification/models/litv2.py","file_url":"https://github.com/ziplab/LITv2/blob/HEAD/classification/models/litv2.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"d9590a53c9850a4d"}},{"code_sha256_prefix":"4673440c036f73eb","entry":"bboxes_iou","repo":"zhuang-group/lit","repo_kind":"listed","path":"classification/code_for_lit_s_m_b/mm_modules/utils.py","file_url":"https://github.com/zhuang-group/lit/blob/HEAD/classification/code_for_lit_s_m_b/mm_modules/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"4673440c036f73eb"}},{"code_sha256_prefix":"45732831d1539a53","entry":"build_model","repo":"zhuang-group/lit","repo_kind":"listed","path":"classification/code_for_lit_s_m_b/models/build.py","file_url":"https://github.com/zhuang-group/lit/blob/HEAD/classification/code_for_lit_s_m_b/models/build.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"45732831d1539a53"}},{"code_sha256_prefix":"c02cc4e0b5720211","entry":"build_scheduler","repo":"zhuang-group/lit","repo_kind":"listed","path":"classification/code_for_lit_s_m_b/lr_scheduler.py","file_url":"https://github.com/zhuang-group/lit/blob/HEAD/classification/code_for_lit_s_m_b/lr_scheduler.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"c02cc4e0b5720211"}},{"code_sha256_prefix":"25517cf1e6458c39","entry":"conv_flops","repo":"zhuang-group/lit","repo_kind":"listed","path":"classification/code_for_lit_s_m_b/models/attentions.py","file_url":"https://github.com/zhuang-group/lit/blob/HEAD/classification/code_for_lit_s_m_b/models/attentions.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"25517cf1e6458c39"}},{"code_sha256_prefix":"80450600f7f09b0a","entry":"create_logger","repo":"zhuang-group/lit","repo_kind":"listed","path":"classification/code_for_lit_s_m_b/logger.py","file_url":"https://github.com/zhuang-group/lit/blob/HEAD/classification/code_for_lit_s_m_b/logger.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"80450600f7f09b0a"}},{"code_sha256_prefix":"02a9e80c2bb81815","entry":"get_config","repo":"zhuang-group/lit","repo_kind":"listed","path":"classification/code_for_lit_s_m_b/config.py","file_url":"https://github.com/zhuang-group/lit/blob/HEAD/classification/code_for_lit_s_m_b/config.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"02a9e80c2bb81815"}},{"code_sha256_prefix":"f1f823695040297a","entry":"matrix_iou","repo":"zhuang-group/lit","repo_kind":"listed","path":"classification/code_for_lit_s_m_b/mm_modules/utils.py","file_url":"https://github.com/zhuang-group/lit/blob/HEAD/classification/code_for_lit_s_m_b/mm_modules/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"f1f823695040297a"}},{"code_sha256_prefix":"9b8d355984e33a51","entry":"postprocess","repo":"zhuang-group/lit","repo_kind":"listed","path":"classification/code_for_lit_s_m_b/mm_modules/utils.py","file_url":"https://github.com/zhuang-group/lit/blob/HEAD/classification/code_for_lit_s_m_b/mm_modules/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"9b8d355984e33a51"}},{"code_sha256_prefix":"9b8aa876c71b0665","entry":"reduce_loss_dict","repo":"zhuang-group/lit","repo_kind":"listed","path":"classification/code_for_lit_s_m_b/mm_modules/distributed_util.py","file_url":"https://github.com/zhuang-group/lit/blob/HEAD/classification/code_for_lit_s_m_b/mm_modules/distributed_util.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"9b8aa876c71b0665"}},{"code_sha256_prefix":"a98b48e2e2be4ce8","entry":"scatter_gather","repo":"zhuang-group/lit","repo_kind":"listed","path":"classification/code_for_lit_s_m_b/mm_modules/distributed_util.py","file_url":"https://github.com/zhuang-group/lit/blob/HEAD/classification/code_for_lit_s_m_b/mm_modules/distributed_util.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"a98b48e2e2be4ce8"}},{"code_sha256_prefix":"29413e543e79fad8","entry":"vis","repo":"zhuang-group/lit","repo_kind":"listed","path":"classification/code_for_lit_s_m_b/mm_modules/vis_utils.py","file_url":"https://github.com/zhuang-group/lit/blob/HEAD/classification/code_for_lit_s_m_b/mm_modules/vis_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"29413e543e79fad8"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}