{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/bridging-the-gap-between-vision-transformers","title":"Bridging the Gap Between Vision Transformers and Convolutional Neural Networks on Small Datasets","arxiv_id":"2210.05958","date":"2022-10-12","proceeding":null,"authors":["Zhiying Lu","Hongtao Xie","Chuanbin Liu","Yongdong Zhang"],"abstract":"There still remains an extreme performance gap between Vision Transformers (ViTs) and Convolutional Neural Networks (CNNs) when training from scratch on small datasets, which is concluded to the lack of inductive bias. In this paper, we further consider this problem and point out two weaknesses of ViTs in inductive biases, that is, the spatial relevance and diverse channel representation. First, on spatial aspect, objects are locally compact and relevant, thus fine-grained feature needs to be extracted from a token and its neighbors. While the lack of data hinders ViTs to attend the spatial relevance. Second, on channel aspect, representation exhibits diversity on different channels. But the scarce data can not enable ViTs to learn strong enough representation for accurate recognition. To this end, we propose Dynamic Hybrid Vision Transformer (DHVT) as the solution to enhance the two inductive biases. On spatial aspect, we adopt a hybrid structure, in which convolution is integrated into patch embedding and multi-layer perceptron module, forcing the model to capture the token features as well as their neighboring features. On channel aspect, we introduce a dynamic feature aggregation module in MLP and a brand new \"head token\" design in multi-head self-attention module to help re-calibrate channel representation and make different channel group representation interacts with each other. The fusion of weak channel representation forms a strong enough representation for classification. With this design, we successfully eliminate the performance gap between CNNs and ViTs, and our DHVT achieves a series of state-of-the-art performance with a lightweight model, 85.68% on CIFAR-100 with 22.8M parameters, 82.3% on ImageNet-1K with 24.0M parameters. Code is available at https://github.com/ArieSeirack/DHVT.","url_abs":"https://arxiv.org/abs/2210.05958v2","url_pdf":"https://arxiv.org/pdf/2210.05958v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"bridging-the-gap-between-vision-transformers","repo_url":"https://github.com/arieseirack/dhvt","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"inductive-bias","task_name":"Inductive Bias"}],"methods":[{"method_slug":"absolute-position-encodings","method_name":"Absolute Position Encodings"},{"method_slug":"adam","method_name":"Adam"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"bpe","method_name":"BPE"},{"method_slug":"convolution","method_name":"Convolution"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"label-smoothing","method_name":"Label Smoothing"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"position-wise-feed-forward-layer","method_name":"Position-Wise Feed-Forward Layer"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"transformer","method_name":"Transformer"},{"method_slug":"vision-transformer","method_name":"Vision Transformer"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2210.05958","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2210.05958"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/arieseirack/dhvt","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran":2,"ran_fixture":2,"unverified":6},"by_repo_kind":{"official":{"samples":10,"ran":4,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"66733f50aff69220","entry":"adjust_keep_rate","repo":"arieseirack/dhvt","repo_kind":"official","path":"helpers.py","file_url":"https://github.com/arieseirack/dhvt/blob/HEAD/helpers.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"66733f50aff69220"}},{"code_sha256_prefix":"3ac6b7d76e8e3584","entry":"drop_path","repo":"arieseirack/dhvt","repo_kind":"official","path":"vision_transformer.py","file_url":"https://github.com/arieseirack/dhvt/blob/HEAD/vision_transformer.py","link_basis":"harvester_set","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"3ac6b7d76e8e3584"}},{"code_sha256_prefix":"191b2735955d18f4","entry":"resize_pos_embed","repo":"arieseirack/dhvt","repo_kind":"official","path":"vision_transformer.py","file_url":"https://github.com/arieseirack/dhvt/blob/HEAD/vision_transformer.py","link_basis":"harvester_set","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"191b2735955d18f4"}},{"code_sha256_prefix":"e294b62d47bcc26f","entry":"speed_test","repo":"arieseirack/dhvt","repo_kind":"official","path":"helpers.py","file_url":"https://github.com/arieseirack/dhvt/blob/HEAD/helpers.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"e294b62d47bcc26f"}},{"code_sha256_prefix":"7085cfd021be27e3","entry":"DiversityLoss","repo":"arieseirack/dhvt","repo_kind":"official","path":"losses.py","file_url":"https://github.com/arieseirack/dhvt/blob/HEAD/losses.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"7085cfd021be27e3"}},{"code_sha256_prefix":"3bc74137ab36aa79","entry":"build_transform","repo":"arieseirack/dhvt","repo_kind":"official","path":"datasets.py","file_url":"https://github.com/arieseirack/dhvt/blob/HEAD/datasets.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"3bc74137ab36aa79"}},{"code_sha256_prefix":"429abe9dc8793925","entry":"conv3x3","repo":"arieseirack/dhvt","repo_kind":"official","path":"vision_transformer.py","file_url":"https://github.com/arieseirack/dhvt/blob/HEAD/vision_transformer.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"429abe9dc8793925"}},{"code_sha256_prefix":"9e7bf674714e06d5","entry":"dumps_pyarrow","repo":"arieseirack/dhvt","repo_kind":"official","path":"folder2lmdb.py","file_url":"https://github.com/arieseirack/dhvt/blob/HEAD/folder2lmdb.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"9e7bf674714e06d5"}},{"code_sha256_prefix":"b4c154b6f0568b6b","entry":"loads_pyarrow","repo":"arieseirack/dhvt","repo_kind":"official","path":"folder2lmdb.py","file_url":"https://github.com/arieseirack/dhvt/blob/HEAD/folder2lmdb.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"b4c154b6f0568b6b"}},{"code_sha256_prefix":"51738010a21f76a1","entry":"raw_reader","repo":"arieseirack/dhvt","repo_kind":"official","path":"folder2lmdb.py","file_url":"https://github.com/arieseirack/dhvt/blob/HEAD/folder2lmdb.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"51738010a21f76a1"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}