{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/designing-bert-for-convolutional-networks","title":"Designing BERT for Convolutional Networks: Sparse and Hierarchical Masked Modeling","arxiv_id":"2301.03580","date":"2023-01-09","proceeding":null,"authors":["Keyu Tian","Yi Jiang","Qishuai Diao","Chen Lin","LiWei Wang","Zehuan Yuan"],"abstract":"We identify and overcome two key obstacles in extending the success of BERT-style pre-training, or the masked image modeling, to convolutional networks (convnets): (i) convolution operation cannot handle irregular, random-masked input images; (ii) the single-scale nature of BERT pre-training is inconsistent with convnet's hierarchical structure. For (i), we treat unmasked pixels as sparse voxels of 3D point clouds and use sparse convolution to encode. This is the first use of sparse convolution for 2D masked modeling. For (ii), we develop a hierarchical decoder to reconstruct images from multi-scale encoded features. Our method called Sparse masKed modeling (SparK) is general: it can be used directly on any convolutional model without backbone modifications. We validate it on both classical (ResNet) and modern (ConvNeXt) models: on three downstream tasks, it surpasses both state-of-the-art contrastive learning and transformer-based masked modeling by similarly large margins (around +1.0%). Improvements on object detection and instance segmentation are more substantial (up to +3.5%), verifying the strong transferability of features learned. We also find its favorable scaling behavior by observing more gains on larger models. All this evidence reveals a promising future of generative pre-training on convnets. Codes and models are released at https://github.com/keyu-tian/SparK.","url_abs":"https://arxiv.org/abs/2301.03580v2","url_pdf":"https://arxiv.org/pdf/2301.03580v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"designing-bert-for-convolutional-networks","repo_url":"https://github.com/keyu-tian/spark","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"designing-bert-for-convolutional-networks","repo_url":"https://github.com/Factral/UnMix-NeRF","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"jax","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"2d-object-detection","task_name":"2D Object Detection"},{"task_slug":"contrastive-learning","task_name":"Contrastive Learning"},{"task_slug":"decoder","task_name":"Decoder"},{"task_slug":"image-classification","task_name":"Image Classification"},{"task_slug":"instance-segmentation","task_name":"Instance Segmentation"},{"task_slug":"object-detection","task_name":"Object Detection"},{"task_slug":"self-supervised-image-classification","task_name":"Self-Supervised Image Classification"},{"task_slug":"semantic-segmentation","task_name":"Semantic Segmentation"},{"task_slug":"object-detection-1","task_name":"object-detection"}],"methods":[{"method_slug":"adam","method_name":"Adam"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"attention-dropout","method_name":"Attention Dropout"},{"method_slug":"bert","method_name":"BERT"},{"method_slug":"contrastive-learning","method_name":"Contrastive Learning"},{"method_slug":"convolution","method_name":"Convolution"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"linear-warmup-with-linear-decay","method_name":"Linear Warmup With Linear Decay"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"weight-decay","method_name":"Weight Decay"},{"method_slug":"wordpiece","method_name":"WordPiece"}],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/image-classification-on-imagenet","task":"Image Classification","dataset":"ImageNet","model":"SparK (ConvNeXt-Large, 384)","rank_in_archive_order":183,"of":1060,"metrics":{"Number of params":"198M","Top 1 Accuracy":"86.0%"},"uses_additional_data":false},{"leaderboard":"/sota/instance-segmentation-on-coco-2017-val","task":"Instance Segmentation","dataset":"COCO 2017 val","model":"SparK (ConvNeXt V1-B Mask R-CNN)","rank_in_archive_order":1,"of":4,"metrics":{"AP":"45.1","mask AP":"45.1","mask AP*":"45.1"},"uses_additional_data":false},{"leaderboard":"/sota/self-supervised-image-classification-on-1","task":"Self-Supervised Image Classification","dataset":"ImageNet (finetuned)","model":"SparK (ConvNeXt-Large, 384)","rank_in_archive_order":18,"of":65,"metrics":{"Number of Params":"198M","Top 1 Accuracy":"86.0%"},"uses_additional_data":false},{"leaderboard":"/sota/self-supervised-image-classification-on-1","task":"Self-Supervised Image Classification","dataset":"ImageNet (finetuned)","model":"SparK (ConvNeXt-Large)","rank_in_archive_order":23,"of":65,"metrics":{"Number of Params":"198M","Top 1 Accuracy":"85.4%"},"uses_additional_data":false},{"leaderboard":"/sota/self-supervised-image-classification-on-1","task":"Self-Supervised Image Classification","dataset":"ImageNet (finetuned)","model":"ConvNeXt-Base (SparK pre-training)","rank_in_archive_order":28,"of":65,"metrics":{"Number of Params":"89M","Top 1 Accuracy":"84.8%"},"uses_additional_data":false},{"leaderboard":"/sota/self-supervised-image-classification-on-1","task":"Self-Supervised Image Classification","dataset":"ImageNet (finetuned)","model":"ConvNeXt-Small (SparK pre-training)","rank_in_archive_order":37,"of":65,"metrics":{"Number of Params":"50M","Top 1 Accuracy":"84.1%"},"uses_additional_data":false},{"leaderboard":"/sota/self-supervised-image-classification-on-1","task":"Self-Supervised Image Classification","dataset":"ImageNet (finetuned)","model":"ResNet-200 (SparK pre-training)","rank_in_archive_order":47,"of":65,"metrics":{"Number of Params":"65M","Top 1 Accuracy":"83.1%"},"uses_additional_data":false},{"leaderboard":"/sota/self-supervised-image-classification-on-1","task":"Self-Supervised Image Classification","dataset":"ImageNet (finetuned)","model":"ResNet-152 (SparK pre-training)","rank_in_archive_order":49,"of":65,"metrics":{"Number of Params":"60M","Top 1 Accuracy":"82.7%"},"uses_additional_data":false},{"leaderboard":"/sota/self-supervised-image-classification-on-1","task":"Self-Supervised Image Classification","dataset":"ImageNet (finetuned)","model":"ResNet-101 (SparK pre-training)","rank_in_archive_order":52,"of":65,"metrics":{"Number of Params":"44M","Top 1 Accuracy":"82.2%"},"uses_additional_data":false},{"leaderboard":"/sota/self-supervised-image-classification-on-1","task":"Self-Supervised Image Classification","dataset":"ImageNet (finetuned)","model":"ResNet-50 (SparK pre-training)","rank_in_archive_order":55,"of":65,"metrics":{"Number of Params":"26M","Top 1 Accuracy":"80.6%"},"uses_additional_data":false}],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2301.03580","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2301.03580"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/Factral/UnMix-NeRF","reach":{"status":"ok","spdx":"Apache-2.0"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/keyu-tian/spark","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":4,"ran_draft_wrong":1,"unverified":9},"by_repo_kind":{"official":{"samples":10,"ran":5,"repositories":1},"listed":{"samples":4,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"aa8d49041d9f8442","entry":"get_num_layer_layer_wise","repo":"keyu-tian/spark","repo_kind":"official","path":"downstream_mmdet/mmcv_custom/layer_decay_optimizer_constructor.py","file_url":"https://github.com/keyu-tian/spark/blob/HEAD/downstream_mmdet/mmcv_custom/layer_decay_optimizer_constructor.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"aa8d49041d9f8442"}},{"code_sha256_prefix":"be134d6ef63b5192","entry":"get_num_layer_stage_wise","repo":"keyu-tian/spark","repo_kind":"official","path":"downstream_mmdet/mmcv_custom/layer_decay_optimizer_constructor.py","file_url":"https://github.com/keyu-tian/spark/blob/HEAD/downstream_mmdet/mmcv_custom/layer_decay_optimizer_constructor.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"be134d6ef63b5192"}},{"code_sha256_prefix":"8733e0d87f232d11","entry":"mixup_target","repo":"keyu-tian/spark","repo_kind":"official","path":"downstream_imagenet/mixup.py","file_url":"https://github.com/keyu-tian/spark/blob/HEAD/downstream_imagenet/mixup.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"8733e0d87f232d11"}},{"code_sha256_prefix":"4728dc2025227bba","entry":"one_hot","repo":"keyu-tian/spark","repo_kind":"official","path":"downstream_imagenet/mixup.py","file_url":"https://github.com/keyu-tian/spark/blob/HEAD/downstream_imagenet/mixup.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"4728dc2025227bba"}},{"code_sha256_prefix":"3913c620f6926f42","entry":"rand_bbox","repo":"keyu-tian/spark","repo_kind":"official","path":"downstream_imagenet/mixup.py","file_url":"https://github.com/keyu-tian/spark/blob/HEAD/downstream_imagenet/mixup.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"3913c620f6926f42"}},{"code_sha256_prefix":"a883944500f3f33d","entry":"component_x","repo":"Factral/UnMix-NeRF","repo_kind":"listed","path":"unmixnerf/utils/spec_to_rgb.py","file_url":"https://github.com/Factral/UnMix-NeRF/blob/HEAD/unmixnerf/utils/spec_to_rgb.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"a883944500f3f33d"}},{"code_sha256_prefix":"8da57cf56b2fdcf1","entry":"component_y","repo":"Factral/UnMix-NeRF","repo_kind":"listed","path":"unmixnerf/utils/spec_to_rgb.py","file_url":"https://github.com/Factral/UnMix-NeRF/blob/HEAD/unmixnerf/utils/spec_to_rgb.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"8da57cf56b2fdcf1"}},{"code_sha256_prefix":"0e865e01e71cf326","entry":"create_classification_dataset","repo":"keyu-tian/spark","repo_kind":"official","path":"downstream_imagenet/data.py","file_url":"https://github.com/keyu-tian/spark/blob/HEAD/downstream_imagenet/data.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"0e865e01e71cf326"}},{"code_sha256_prefix":"558f6d7e47654006","entry":"g","repo":"Factral/UnMix-NeRF","repo_kind":"listed","path":"unmixnerf/utils/spec_to_rgb.py","file_url":"https://github.com/Factral/UnMix-NeRF/blob/HEAD/unmixnerf/utils/spec_to_rgb.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"558f6d7e47654006"}},{"code_sha256_prefix":"ad373c0953d230ed","entry":"get_param_groups","repo":"keyu-tian/spark","repo_kind":"official","path":"downstream_imagenet/lr_decay.py","file_url":"https://github.com/keyu-tian/spark/blob/HEAD/downstream_imagenet/lr_decay.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ad373c0953d230ed"}},{"code_sha256_prefix":"36ae228f14629970","entry":"get_weights_spectral","repo":"Factral/UnMix-NeRF","repo_kind":"listed","path":"unmixnerf/unmix_renderer.py","file_url":"https://github.com/Factral/UnMix-NeRF/blob/HEAD/unmixnerf/unmix_renderer.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"36ae228f14629970"}},{"code_sha256_prefix":"7475c50600ea1392","entry":"load_checkpoint","repo":"keyu-tian/spark","repo_kind":"official","path":"downstream_imagenet/util.py","file_url":"https://github.com/keyu-tian/spark/blob/HEAD/downstream_imagenet/util.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"7475c50600ea1392"}},{"code_sha256_prefix":"800af14f9c740324","entry":"lr_factor_func","repo":"keyu-tian/spark","repo_kind":"official","path":"downstream_d2/lr_decay.py","file_url":"https://github.com/keyu-tian/spark/blob/HEAD/downstream_d2/lr_decay.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"800af14f9c740324"}},{"code_sha256_prefix":"f921a5666329b4eb","entry":"lr_wd_annealing","repo":"keyu-tian/spark","repo_kind":"official","path":"downstream_imagenet/lr_decay.py","file_url":"https://github.com/keyu-tian/spark/blob/HEAD/downstream_imagenet/lr_decay.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f921a5666329b4eb"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}