{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/the-principle-of-diversity-training-stronger","title":"The Principle of Diversity: Training Stronger Vision Transformers Calls for Reducing All Levels of Redundancy","arxiv_id":"2203.06345","date":"2022-03-12","proceeding":"CVPR 2022 1","authors":["Tianlong Chen","Zhenyu Zhang","Yu Cheng","Ahmed Awadallah","Zhangyang Wang"],"abstract":"Vision transformers (ViTs) have gained increasing popularity as they are commonly believed to own higher modeling capacity and representation flexibility, than traditional convolutional networks. However, it is questionable whether such potential has been fully unleashed in practice, as the learned ViTs often suffer from over-smoothening, yielding likely redundant models. Recent works made preliminary attempts to identify and alleviate such redundancy, e.g., via regularizing embedding similarity or re-injecting convolution-like structures. However, a \"head-to-toe assessment\" regarding the extent of redundancy in ViTs, and how much we could gain by thoroughly mitigating such, has been absent for this field. This paper, for the first time, systematically studies the ubiquitous existence of redundancy at all three levels: patch embedding, attention map, and weight space. In view of them, we advocate a principle of diversity for training ViTs, by presenting corresponding regularizers that encourage the representation diversity and coverage at each of those levels, that enabling capturing more discriminative information. Extensive experiments on ImageNet with a number of ViT backbones validate the effectiveness of our proposals, largely eliminating the observed ViT redundancy and significantly boosting the model generalization. For example, our diversified DeiT obtains 0.70%~1.76% accuracy boosts on ImageNet with highly reduced similarity. Our codes are fully available in https://github.com/VITA-Group/Diverse-ViT.","url_abs":"https://arxiv.org/abs/2203.06345v1","url_pdf":"https://arxiv.org/pdf/2203.06345v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"the-principle-of-diversity-training-stronger","repo_url":"https://github.com/vita-group/diverse-vit","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"all","task_name":"All"},{"task_slug":"diversity","task_name":"Diversity"}],"methods":[{"method_slug":"attention","method_name":"Attention"},{"method_slug":"attention-dropout","method_name":"Attention Dropout"},{"method_slug":"deit","method_name":"DeiT"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"feedforward-network","method_name":"Feedforward Network"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"softmax","method_name":"Softmax"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2203.06345","atlas_url":"https://app.syntology.ai/?focus=2203.06345","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2203.06345"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/vita-group/diverse-vit","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"deterministic:regex_extraction","url":"https://github.com/VITA-Group/Diverse-ViT","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":9,"ran_fixture":1,"unverified":3},"by_repo_kind":{"official":{"samples":13,"ran":10,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"917f7c598d2d4810","entry":"Loss_contrastive","repo":"VITA-Group/Diverse-ViT","repo_kind":"official","path":"reg.py","file_url":"https://github.com/VITA-Group/Diverse-ViT/blob/HEAD/reg.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"917f7c598d2d4810"}},{"code_sha256_prefix":"9d7670f2c022eca9","entry":"Loss_cosine","repo":"VITA-Group/Diverse-ViT","repo_kind":"official","path":"reg.py","file_url":"https://github.com/VITA-Group/Diverse-ViT/blob/HEAD/reg.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"9d7670f2c022eca9"}},{"code_sha256_prefix":"146e4615af0a0d1f","entry":"Loss_mixing","repo":"VITA-Group/Diverse-ViT","repo_kind":"official","path":"reg.py","file_url":"https://github.com/VITA-Group/Diverse-ViT/blob/HEAD/reg.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"146e4615af0a0d1f"}},{"code_sha256_prefix":"c1e69f1ba76b4145","entry":"get_batch","repo":"VITA-Group/Diverse-ViT","repo_kind":"official","path":"gradient_utils.py","file_url":"https://github.com/VITA-Group/Diverse-ViT/blob/HEAD/gradient_utils.py","link_basis":"harvester_set","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c1e69f1ba76b4145"}},{"code_sha256_prefix":"a04f6c217a55c7c9","entry":"get_ordered_params","repo":"VITA-Group/Diverse-ViT","repo_kind":"official","path":"gradient_utils.py","file_url":"https://github.com/VITA-Group/Diverse-ViT/blob/HEAD/gradient_utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a04f6c217a55c7c9"}},{"code_sha256_prefix":"61109833276c1a1a","entry":"get_scale_stats","repo":"VITA-Group/Diverse-ViT","repo_kind":"official","path":"gradient_utils.py","file_url":"https://github.com/VITA-Group/Diverse-ViT/blob/HEAD/gradient_utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"61109833276c1a1a"}},{"code_sha256_prefix":"b593bfa372a89274","entry":"mixup_target","repo":"VITA-Group/Diverse-ViT","repo_kind":"official","path":"mix.py","file_url":"https://github.com/VITA-Group/Diverse-ViT/blob/HEAD/mix.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b593bfa372a89274"}},{"code_sha256_prefix":"a1e046d46f0b9ca0","entry":"one_hot","repo":"VITA-Group/Diverse-ViT","repo_kind":"official","path":"mix.py","file_url":"https://github.com/VITA-Group/Diverse-ViT/blob/HEAD/mix.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a1e046d46f0b9ca0"}},{"code_sha256_prefix":"d0b649ce5448ea07","entry":"rand_bbox","repo":"VITA-Group/Diverse-ViT","repo_kind":"official","path":"mix.py","file_url":"https://github.com/VITA-Group/Diverse-ViT/blob/HEAD/mix.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d0b649ce5448ea07"}},{"code_sha256_prefix":"743923947c250e6c","entry":"resize_pos_embed","repo":"VITA-Group/Diverse-ViT","repo_kind":"official","path":"vision_transformer_diverse.py","file_url":"https://github.com/VITA-Group/Diverse-ViT/blob/HEAD/vision_transformer_diverse.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"743923947c250e6c"}},{"code_sha256_prefix":"49c67c60672b93ae","entry":"build_dataset","repo":"VITA-Group/Diverse-ViT","repo_kind":"official","path":"datasets.py","file_url":"https://github.com/VITA-Group/Diverse-ViT/blob/HEAD/datasets.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"49c67c60672b93ae"}},{"code_sha256_prefix":"3bc74137ab36aa79","entry":"build_transform","repo":"VITA-Group/Diverse-ViT","repo_kind":"official","path":"datasets.py","file_url":"https://github.com/VITA-Group/Diverse-ViT/blob/HEAD/datasets.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"3bc74137ab36aa79"}},{"code_sha256_prefix":"df40006af8733223","entry":"checkpoint_filter_fn","repo":"VITA-Group/Diverse-ViT","repo_kind":"official","path":"vision_transformer_diverse.py","file_url":"https://github.com/VITA-Group/Diverse-ViT/blob/HEAD/vision_transformer_diverse.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"df40006af8733223"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}