{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/re-masked-autoencoders-are-small-scale-vision","title":"[Re] Masked Autoencoders Are Small Scale Vision Learners: A Reproduction Under Resource Constraints","arxiv_id":null,"date":"2023-09-21","proceeding":"NeurIPS 2023 11","authors":[],"abstract":"Scope of Reproducibility — The Masked Autoencoder (MAE) was recently proposed as a\nframework for efficient self‐supervised pre‐training in Computer Vision [1]. In this pa‐\nper, we attempt a replication of the MAE under significant computational constraints.\nSpecifically, we target the claim that masking out a large part of the input image yields\na nontrivial and meaningful self‐supervisory task, which allows training models that\ngeneralize well. We also present the Semantic Masked Autoencoder (SMAE), a novel yet\nsimple extension of MAE which uses perceptual loss to improve encoder embeddings.\n\nMethodology — The datasets and backbones we rely on are significantly smaller than those\nused by [1]. Our main experiments are performed on Tiny ImageNet (TIN) [2] and trans‐\nfer learning is performed on a low‐resolution version of CUB‐200‐2011 [3]. We use a\nViT‐Lite [4] as backbone. We also compare the MAE to DINO, an alternative frame‐\nwork for self‐supervised learning [5]. The ViT, MAE, as well as perceptual loss were\nimplemented from scratch, without consulting the original authors’ code. Our code is\navailable at https://github.com/MLReproHub/SMAE. The computational budget for our\nreproduction and extension was approximately 150 GPU hours.\n\nResults — This paper successfully reproduces the claim that the MAE poses a nontrivial\nand meaningful self‐supervisory task. We show that models trained with this frame‐\nwork generalize well to new datasets and conclude that the MAE is reproducible with\nexception for some hyperparameter choices. We also demonstrate that MAE performs\nwell with smaller backbones and datasets. Finally, our results suggest that the SMAE\nextension improves the downstream classification accuracy of the MAE on CUB (+5 pp)\nwhen coupled with an appropriate masking strategy.\n\nWhat was easy — Given prior experience with a deep learning framework, re‐implementing\nthe paper was relatively straightforward, with sufficient details given in the paper.\n\nWhat was difficult — We faced challenges implementing efficient patch shuffling and tun‐\ning hyperparameters. The hyperparameter choices from [1] did not translate well to a\nsmaller dataset and backbone.\n\nCommunication with original authors — We have not had contact with the original authors.Paper Url: https://openaccess.thecvf.com/content/CVPR2022/html/He_Masked_Autoencoders_Are_Scalable_Vision_Learners_CVPR_2022_paper.htmlPaper Venue: CVPR 2022Confirmation: The report pdf is generated from the provided camera ready Google Colab script, The report metadata is verified from the camera ready Google Colab script, The report contains correct author information., The report contains link to code and SWH metadata., The report follows the ReScience latex style guides as in the Reproducibility Report Template (https://paperswithcode.com/rc2022/registration)., The report contains the Reproducibility Summary in the first page., The latex .zip file is verified from the camera ready Google Colab scriptLatex:  zipJournal: ReScience Volume 9 Issue 2 Article 40Doi: https://www.doi.org/10.5281/zenodo.8173751Code: https://archive.softwareheritage.org/swh:1:dir:4d37d466bafc5dc45bf5ba68caa53f207e6d0702","url_abs":"https://openreview.net/forum?id=KXfjZPL5pqr","url_pdf":"https://openreview.net/pdf?id=FasIQqsJhe","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"re-masked-autoencoders-are-small-scale-vision","repo_url":"https://github.com/mlreprohub/smae","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":null}],"tasks":[],"methods":[{"method_slug":"attention","method_name":"Attention"},{"method_slug":"dino","method_name":"DINO"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"mae","method_name":"MAE"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"vision-transformer","method_name":"Vision Transformer"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":null,"mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}