{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/improving-vision-transformers-by-revisiting","title":"Improving Vision Transformers by Revisiting High-frequency Components","arxiv_id":"2204.00993","date":"2022-04-03","proceeding":null,"authors":["Jiawang Bai","Li Yuan","Shu-Tao Xia","Shuicheng Yan","Zhifeng Li","Wei Liu"],"abstract":"The transformer models have shown promising effectiveness in dealing with various vision tasks. However, compared with training Convolutional Neural Network (CNN) models, training Vision Transformer (ViT) models is more difficult and relies on the large-scale training set. To explain this observation we make a hypothesis that \\textit{ViT models are less effective in capturing the high-frequency components of images than CNN models}, and verify it by a frequency analysis. Inspired by this finding, we first investigate the effects of existing techniques for improving ViT models from a new frequency perspective, and find that the success of some techniques (e.g., RandAugment) can be attributed to the better usage of the high-frequency components. Then, to compensate for this insufficient ability of ViT models, we propose HAT, which directly augments high-frequency components of images via adversarial training. We show that HAT can consistently boost the performance of various ViT models (e.g., +1.2% for ViT-B, +0.5% for Swin-B), and especially enhance the advanced model VOLO-D5 to 87.3% that only uses ImageNet-1K data, and the superiority can also be maintained on out-of-distribution data and transferred to downstream tasks. The code is available at: https://github.com/jiawangbai/HAT.","url_abs":"https://arxiv.org/abs/2204.00993v3","url_pdf":"https://arxiv.org/pdf/2204.00993v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"improving-vision-transformers-by-revisiting","repo_url":"https://github.com/jiawangbai/HAT","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"domain-generalization","task_name":"Domain Generalization"},{"task_slug":"image-classification","task_name":"Image Classification"},{"task_slug":"high","task_name":"Vocal Bursts Intensity Prediction"}],"methods":[{"method_slug":"absolute-position-encodings","method_name":"Absolute Position Encodings"},{"method_slug":"adam","method_name":"Adam"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"bpe","method_name":"BPE"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"label-smoothing","method_name":"Label Smoothing"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"position-wise-feed-forward-layer","method_name":"Position-Wise Feed-Forward Layer"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"transformer","method_name":"Transformer"},{"method_slug":"vision-transformer","method_name":"Vision Transformer"}],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/domain-generalization-on-imagenet-c","task":"Domain Generalization","dataset":"ImageNet-C","model":"VOLO-D5+HAT","rank_in_archive_order":10,"of":47,"metrics":{"Number of params":"296M","mean Corruption Error (mCE)":"38.4"},"uses_additional_data":false},{"leaderboard":"/sota/domain-generalization-on-imagenet-r","task":"Domain Generalization","dataset":"ImageNet-R","model":"VOLO-D5+HAT","rank_in_archive_order":16,"of":39,"metrics":{"Top-1 Error Rate":"40.3"},"uses_additional_data":false},{"leaderboard":"/sota/domain-generalization-on-stylized-imagenet","task":"Domain Generalization","dataset":"Stylized-ImageNet","model":"VOLO-D5+HAT","rank_in_archive_order":2,"of":3,"metrics":{"Top 1 Accuracy":"25.9"},"uses_additional_data":false},{"leaderboard":"/sota/image-classification-on-imagenet","task":"Image Classification","dataset":"ImageNet","model":"VOLO-D5+HAT","rank_in_archive_order":98,"of":1060,"metrics":{"GFLOPs":"412","Number of params":"295.5M","Top 1 Accuracy":"87.3%"},"uses_additional_data":false}],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2204.00993","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2204.00993"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/jiawangbai/HAT","reach":null}],"summary":{"ran":10,"unverified":4},"by_repo_kind":{"official":{"samples":14,"ran":10,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"758bd44a54df5709","entry":"Attention","repo":"jiawangbai/HAT","repo_kind":"official","path":"models/volo.py","file_url":"https://github.com/jiawangbai/HAT/blob/HEAD/models/volo.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"758bd44a54df5709"}},{"code_sha256_prefix":"d6aec4de1a988995","entry":"ClassAttention","repo":"jiawangbai/HAT","repo_kind":"official","path":"models/volo.py","file_url":"https://github.com/jiawangbai/HAT/blob/HEAD/models/volo.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d6aec4de1a988995"}},{"code_sha256_prefix":"b10f308ac343586f","entry":"ClassBlock","repo":"jiawangbai/HAT","repo_kind":"official","path":"models/volo.py","file_url":"https://github.com/jiawangbai/HAT/blob/HEAD/models/volo.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b10f308ac343586f"}},{"code_sha256_prefix":"33ecaec51f88c518","entry":"Downsample","repo":"jiawangbai/HAT","repo_kind":"official","path":"models/volo.py","file_url":"https://github.com/jiawangbai/HAT/blob/HEAD/models/volo.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"33ecaec51f88c518"}},{"code_sha256_prefix":"f3cd14e6c18a60d8","entry":"Mlp","repo":"jiawangbai/HAT","repo_kind":"official","path":"models/volo.py","file_url":"https://github.com/jiawangbai/HAT/blob/HEAD/models/volo.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f3cd14e6c18a60d8"}},{"code_sha256_prefix":"0b3e7de2ecb2c1e7","entry":"OutlookAttention","repo":"jiawangbai/HAT","repo_kind":"official","path":"models/volo.py","file_url":"https://github.com/jiawangbai/HAT/blob/HEAD/models/volo.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"0b3e7de2ecb2c1e7"}},{"code_sha256_prefix":"53a133709900d494","entry":"PatchEmbed","repo":"jiawangbai/HAT","repo_kind":"official","path":"models/volo.py","file_url":"https://github.com/jiawangbai/HAT/blob/HEAD/models/volo.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"53a133709900d494"}},{"code_sha256_prefix":"f1380508396f9eb9","entry":"Transformer","repo":"jiawangbai/HAT","repo_kind":"official","path":"models/volo.py","file_url":"https://github.com/jiawangbai/HAT/blob/HEAD/models/volo.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f1380508396f9eb9"}},{"code_sha256_prefix":"ba7779c554538760","entry":"outlooker_blocks","repo":"jiawangbai/HAT","repo_kind":"official","path":"models/volo.py","file_url":"https://github.com/jiawangbai/HAT/blob/HEAD/models/volo.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ba7779c554538760"}},{"code_sha256_prefix":"bcb4ce1c1b25bc6c","entry":"transformer_blocks","repo":"jiawangbai/HAT","repo_kind":"official","path":"models/volo.py","file_url":"https://github.com/jiawangbai/HAT/blob/HEAD/models/volo.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"bcb4ce1c1b25bc6c"}},{"code_sha256_prefix":"a114b78025e6c55f","entry":"Outlooker","repo":"jiawangbai/HAT","repo_kind":"official","path":"models/volo.py","file_url":"https://github.com/jiawangbai/HAT/blob/HEAD/models/volo.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a114b78025e6c55f"}},{"code_sha256_prefix":"2b66fd4ced7843b3","entry":"VOLO","repo":"jiawangbai/HAT","repo_kind":"official","path":"models/volo.py","file_url":"https://github.com/jiawangbai/HAT/blob/HEAD/models/volo.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"2b66fd4ced7843b3"}},{"code_sha256_prefix":"6974cb1ca3926a70","entry":"get_block","repo":"jiawangbai/HAT","repo_kind":"official","path":"models/volo.py","file_url":"https://github.com/jiawangbai/HAT/blob/HEAD/models/volo.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"6974cb1ca3926a70"}},{"code_sha256_prefix":"b66088e90761117d","entry":"rand_bbox","repo":"jiawangbai/HAT","repo_kind":"official","path":"models/volo.py","file_url":"https://github.com/jiawangbai/HAT/blob/HEAD/models/volo.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b66088e90761117d"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}