{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/bootstrapped-masked-autoencoders-for-vision","title":"Bootstrapped Masked Autoencoders for Vision BERT Pretraining","arxiv_id":"2207.07116","date":"2022-07-14","proceeding":null,"authors":["Xiaoyi Dong","Jianmin Bao","Ting Zhang","Dongdong Chen","Weiming Zhang","Lu Yuan","Dong Chen","Fang Wen","Nenghai Yu"],"abstract":"We propose bootstrapped masked autoencoders (BootMAE), a new approach for vision BERT pretraining. BootMAE improves the original masked autoencoders (MAE) with two core designs: 1) momentum encoder that provides online feature as extra BERT prediction targets; 2) target-aware decoder that tries to reduce the pressure on the encoder to memorize target-specific information in BERT pretraining. The first design is motivated by the observation that using a pretrained MAE to extract the features as the BERT prediction target for masked tokens can achieve better pretraining performance. Therefore, we add a momentum encoder in parallel with the original MAE encoder, which bootstraps the pretraining performance by using its own representation as the BERT prediction target. In the second design, we introduce target-specific information (e.g., pixel values of unmasked patches) from the encoder directly to the decoder to reduce the pressure on the encoder of memorizing the target-specific information. Thus, the encoder focuses on semantic modeling, which is the goal of BERT pretraining, and does not need to waste its capacity in memorizing the information of unmasked tokens related to the prediction target. Through extensive experiments, our BootMAE achieves $84.2\\%$ Top-1 accuracy on ImageNet-1K with ViT-B backbone, outperforming MAE by $+0.8\\%$ under the same pre-training epochs. BootMAE also gets $+1.0$ mIoU improvements on semantic segmentation on ADE20K and $+1.3$ box AP, $+1.4$ mask AP improvement on object detection and segmentation on COCO dataset. Code is released at https://github.com/LightDXY/BootMAE.","url_abs":"https://arxiv.org/abs/2207.07116v1","url_pdf":"https://arxiv.org/pdf/2207.07116v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"bootstrapped-masked-autoencoders-for-vision","repo_url":"https://github.com/lightdxy/bootmae","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":"decoder","task_name":"Decoder"},{"task_slug":"object-detection","task_name":"Object Detection"},{"task_slug":"prediction","task_name":"Prediction"},{"task_slug":"self-supervised-image-classification","task_name":"Self-Supervised Image Classification"},{"task_slug":"semantic-segmentation","task_name":"Semantic Segmentation"}],"methods":[{"method_slug":"adam","method_name":"Adam"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"attention-dropout","method_name":"Attention Dropout"},{"method_slug":"bert","method_name":"BERT"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"linear-warmup-with-linear-decay","method_name":"Linear Warmup With Linear Decay"},{"method_slug":"mae","method_name":"MAE"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"weight-decay","method_name":"Weight Decay"},{"method_slug":"wordpiece","method_name":"WordPiece"}],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/self-supervised-image-classification-on-1","task":"Self-Supervised Image Classification","dataset":"ImageNet (finetuned)","model":"BootMAE(ViT-L)","rank_in_archive_order":19,"of":65,"metrics":{"Number of Params":"307M","Top 1 Accuracy":"85.9%"},"uses_additional_data":false}],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2207.07116","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2207.07116"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/lightdxy/bootmae","reach":{"status":"ok"}},{"provenance":"deterministic:regex_extraction","url":"https://github.com/LightDXY/BootMAE","reach":{"status":"ok"}}],"summary":{"ran":6,"unverified":1},"by_repo_kind":{"official":{"samples":7,"ran":6,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":7,"samples":[{"code_sha256_prefix":"dc9091be97d3adb7","entry":"Attention","repo":"LightDXY/BootMAE","repo_kind":"official","path":"models/modeling_pretrain_bootmae.py","file_url":"https://github.com/LightDXY/BootMAE/blob/HEAD/models/modeling_pretrain_bootmae.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"dc9091be97d3adb7"}},{"code_sha256_prefix":"2e233dc9eecdb15a","entry":"Block","repo":"LightDXY/BootMAE","repo_kind":"official","path":"models/modeling_pretrain_bootmae.py","file_url":"https://github.com/LightDXY/BootMAE/blob/HEAD/models/modeling_pretrain_bootmae.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"2e233dc9eecdb15a"}},{"code_sha256_prefix":"99e76110604f3828","entry":"Cross_Attention","repo":"LightDXY/BootMAE","repo_kind":"official","path":"models/modeling_pretrain_bootmae.py","file_url":"https://github.com/LightDXY/BootMAE/blob/HEAD/models/modeling_pretrain_bootmae.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"99e76110604f3828"}},{"code_sha256_prefix":"285a20f6856a64df","entry":"Decode_Block","repo":"LightDXY/BootMAE","repo_kind":"official","path":"models/modeling_pretrain_bootmae.py","file_url":"https://github.com/LightDXY/BootMAE/blob/HEAD/models/modeling_pretrain_bootmae.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"285a20f6856a64df"}},{"code_sha256_prefix":"bacdc8a1f921c374","entry":"PatchDeEmbed","repo":"LightDXY/BootMAE","repo_kind":"official","path":"models/modeling_pretrain_bootmae.py","file_url":"https://github.com/LightDXY/BootMAE/blob/HEAD/models/modeling_pretrain_bootmae.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"bacdc8a1f921c374"}},{"code_sha256_prefix":"6eb7118cc6482ba6","entry":"PatchEmbed","repo":"LightDXY/BootMAE","repo_kind":"official","path":"models/modeling_pretrain_bootmae.py","file_url":"https://github.com/LightDXY/BootMAE/blob/HEAD/models/modeling_pretrain_bootmae.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"6eb7118cc6482ba6"}},{"code_sha256_prefix":"420c8a5166a48d7f","entry":"VisionTransformerForMaskedImageModeling","repo":"LightDXY/BootMAE","repo_kind":"official","path":"models/modeling_pretrain_bootmae.py","file_url":"https://github.com/LightDXY/BootMAE/blob/HEAD/models/modeling_pretrain_bootmae.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"420c8a5166a48d7f"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}