{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/depth-wise-convolutions-in-vision","title":"Depth-Wise Convolutions in Vision Transformers for Efficient Training on Small Datasets","arxiv_id":"2407.19394","date":"2024-07-28","proceeding":null,"authors":["Tianxiao Zhang","Wenju Xu","Bo Luo","Guanghui Wang"],"abstract":"The Vision Transformer (ViT) leverages the Transformer's encoder to capture global information by dividing images into patches and achieves superior performance across various computer vision tasks. However, the self-attention mechanism of ViT captures the global context from the outset, overlooking the inherent relationships between neighboring pixels in images or videos. Transformers mainly focus on global information while ignoring the fine-grained local details. Consequently, ViT lacks inductive bias during image or video dataset training. In contrast, convolutional neural networks (CNNs), with their reliance on local filters, possess an inherent inductive bias, making them more efficient and quicker to converge than ViT with less data. In this paper, we present a lightweight Depth-Wise Convolution module as a shortcut in ViT models, bypassing entire Transformer blocks to ensure the models capture both local and global information with minimal overhead. Additionally, we introduce two architecture variants, allowing the Depth-Wise Convolution modules to be applied to multiple Transformer blocks for parameter savings, and incorporating independent parallel Depth-Wise Convolution modules with different kernels to enhance the acquisition of local information. The proposed approach significantly boosts the performance of ViT models on image classification, object detection, and instance segmentation by a large margin, especially on small datasets, as evaluated on CIFAR-10, CIFAR-100, Tiny-ImageNet and ImageNet for image classification, and COCO for object detection and instance segmentation. The source code can be accessed at https://github.com/ZTX-100/Efficient_ViT_with_DW.","url_abs":"https://arxiv.org/abs/2407.19394v4","url_pdf":"https://arxiv.org/pdf/2407.19394v4.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"depth-wise-convolutions-in-vision","repo_url":"https://github.com/ztx-100/efficient_vit_with_dw","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":"image-classification","task_name":"Image Classification"},{"task_slug":"inductive-bias","task_name":"Inductive Bias"},{"task_slug":"instance-segmentation","task_name":"Instance Segmentation"},{"task_slug":"object-detection","task_name":"Object Detection"},{"task_slug":"semantic-segmentation","task_name":"Semantic Segmentation"},{"task_slug":"image-classification","task_name":"image-classification"},{"task_slug":"object-detection-1","task_name":"object-detection"}],"methods":[{"method_slug":"absolute-position-encodings","method_name":"Absolute Position Encodings"},{"method_slug":"adam","method_name":"Adam"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"bpe","method_name":"BPE"},{"method_slug":"convolution","method_name":"Convolution"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"focus","method_name":"Focus"},{"method_slug":"label-smoothing","method_name":"Label Smoothing"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"position-wise-feed-forward-layer","method_name":"Position-Wise Feed-Forward Layer"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"transformer","method_name":"Transformer"},{"method_slug":"vision-transformer","method_name":"Vision Transformer"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":null,"atlas_url":"https://app.syntology.ai/?focus=2407.19394","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2407.19394"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/ztx-100/efficient_vit_with_dw","reach":{"status":"ok"}}],"summary":{"ran_fixture":1,"ran":4,"ran_violates":2,"ran_honours":1,"ran_draft_wrong":1,"unverified":7},"by_repo_kind":{"official":{"samples":16,"ran":9,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":16,"samples":[{"code_sha256_prefix":"144d10b49baeb8a6","entry":"window_partition","repo":"ztx-100/efficient_vit_with_dw","repo_kind":"official","path":"models/swin_transformer.py","file_url":"https://github.com/ztx-100/efficient_vit_with_dw/blob/HEAD/models/swin_transformer.py","link_basis":"harvester_set","language":"python","status":"ran_fixture","verification_level":2,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"144d10b49baeb8a6"}},{"code_sha256_prefix":"9fe330952c4ee761","entry":"build_optimizer","repo":"ztx-100/efficient_vit_with_dw","repo_kind":"official","path":"optimizer.py","file_url":"https://github.com/ztx-100/efficient_vit_with_dw/blob/HEAD/optimizer.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"9fe330952c4ee761"}},{"code_sha256_prefix":"5c177b03eb4157b4","entry":"dropout_layers","repo":"ztx-100/efficient_vit_with_dw","repo_kind":"official","path":"models/cait.py","file_url":"https://github.com/ztx-100/efficient_vit_with_dw/blob/HEAD/models/cait.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"5c177b03eb4157b4"}},{"code_sha256_prefix":"aa5486a3650902d8","entry":"exists","repo":"ztx-100/efficient_vit_with_dw","repo_kind":"official","path":"models/cait.py","file_url":"https://github.com/ztx-100/efficient_vit_with_dw/blob/HEAD/models/cait.py","link_basis":"harvester_set","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"aa5486a3650902d8"}},{"code_sha256_prefix":"eec1e7cba51d5e8e","entry":"get_grad_norm","repo":"ztx-100/efficient_vit_with_dw","repo_kind":"official","path":"utils.py","file_url":"https://github.com/ztx-100/efficient_vit_with_dw/blob/HEAD/utils.py","link_basis":"harvester_set","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"eec1e7cba51d5e8e"}},{"code_sha256_prefix":"59c3a4f0d92e6970","entry":"load_checkpoint","repo":"ztx-100/efficient_vit_with_dw","repo_kind":"official","path":"utils.py","file_url":"https://github.com/ztx-100/efficient_vit_with_dw/blob/HEAD/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"59c3a4f0d92e6970"}},{"code_sha256_prefix":"6ba8cee9f5daea41","entry":"pair","repo":"ztx-100/efficient_vit_with_dw","repo_kind":"official","path":"models/vit.py","file_url":"https://github.com/ztx-100/efficient_vit_with_dw/blob/HEAD/models/vit.py","link_basis":"plan_row","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"well_formed","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"6ba8cee9f5daea41"}},{"code_sha256_prefix":"b33222a09fc93bec","entry":"set_weight_decay","repo":"ztx-100/efficient_vit_with_dw","repo_kind":"official","path":"optimizer.py","file_url":"https://github.com/ztx-100/efficient_vit_with_dw/blob/HEAD/optimizer.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"b33222a09fc93bec"}},{"code_sha256_prefix":"61bf152e6a42a184","entry":"window_reverse","repo":"ztx-100/efficient_vit_with_dw","repo_kind":"official","path":"models/swin_transformer.py","file_url":"https://github.com/ztx-100/efficient_vit_with_dw/blob/HEAD/models/swin_transformer.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"61bf152e6a42a184"}},{"code_sha256_prefix":"0c32459bef92ff8c","entry":"auto_resume_helper","repo":"ztx-100/efficient_vit_with_dw","repo_kind":"official","path":"utils.py","file_url":"https://github.com/ztx-100/efficient_vit_with_dw/blob/HEAD/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"0c32459bef92ff8c"}},{"code_sha256_prefix":"7cc4ee2a64e93062","entry":"build_model","repo":"ztx-100/efficient_vit_with_dw","repo_kind":"official","path":"models/build.py","file_url":"https://github.com/ztx-100/efficient_vit_with_dw/blob/HEAD/models/build.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"7cc4ee2a64e93062"}},{"code_sha256_prefix":"0fbdf3257aea9bcc","entry":"build_scheduler","repo":"ztx-100/efficient_vit_with_dw","repo_kind":"official","path":"lr_scheduler.py","file_url":"https://github.com/ztx-100/efficient_vit_with_dw/blob/HEAD/lr_scheduler.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"0fbdf3257aea9bcc"}},{"code_sha256_prefix":"b0a5beb34716d5a6","entry":"check_keywords_in_name","repo":"ztx-100/efficient_vit_with_dw","repo_kind":"official","path":"optimizer.py","file_url":"https://github.com/ztx-100/efficient_vit_with_dw/blob/HEAD/optimizer.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"b0a5beb34716d5a6"}},{"code_sha256_prefix":"80450600f7f09b0a","entry":"create_logger","repo":"ztx-100/efficient_vit_with_dw","repo_kind":"official","path":"logger.py","file_url":"https://github.com/ztx-100/efficient_vit_with_dw/blob/HEAD/logger.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"80450600f7f09b0a"}},{"code_sha256_prefix":"759a6b3cb8363354","entry":"get_config","repo":"ztx-100/efficient_vit_with_dw","repo_kind":"official","path":"config.py","file_url":"https://github.com/ztx-100/efficient_vit_with_dw/blob/HEAD/config.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"759a6b3cb8363354"}},{"code_sha256_prefix":"a876af529e9074d6","entry":"update_config","repo":"ztx-100/efficient_vit_with_dw","repo_kind":"official","path":"config.py","file_url":"https://github.com/ztx-100/efficient_vit_with_dw/blob/HEAD/config.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"a876af529e9074d6"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}