{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/multimodal-masked-autoencoders-learn","title":"Multimodal Masked Autoencoders Learn Transferable Representations","arxiv_id":"2205.14204","date":"2022-05-27","proceeding":null,"authors":["Xinyang Geng","Hao liu","Lisa Lee","Dale Schuurmans","Sergey Levine","Pieter Abbeel"],"abstract":"Building scalable models to learn from diverse, multimodal data remains an open challenge. For vision-language data, the dominant approaches are based on contrastive learning objectives that train a separate encoder for each modality. While effective, contrastive learning approaches introduce sampling bias depending on the data augmentations used, which can degrade performance on downstream tasks. Moreover, these methods are limited to paired image-text data, and cannot leverage widely-available unpaired data. In this paper, we investigate whether a large multimodal model trained purely via masked token prediction, without using modality-specific encoders or contrastive learning, can learn transferable representations for downstream tasks. We propose a simple and scalable network architecture, the Multimodal Masked Autoencoder (M3AE), which learns a unified encoder for both vision and language data via masked token prediction. We provide an empirical study of M3AE trained on a large-scale image-text dataset, and find that M3AE is able to learn generalizable representations that transfer well to downstream tasks. Surprisingly, we find that M3AE benefits from a higher text mask ratio (50-90%), in contrast to BERT whose standard masking ratio is 15%, due to the joint training of two data modalities. We also provide qualitative analysis showing that the learned representation incorporates meaningful information from both image and language. Lastly, we demonstrate the scalability of M3AE with larger model size and training time, and its flexibility to train on both paired image-text data as well as unpaired data.","url_abs":"https://arxiv.org/abs/2205.14204v3","url_pdf":"https://arxiv.org/pdf/2205.14204v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"multimodal-masked-autoencoders-learn","repo_url":"https://github.com/young-geng/m3ae_public","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"jax","reach":null},{"paper_slug":"multimodal-masked-autoencoders-learn","repo_url":"https://github.com/snoop2head/ehrxqa-2024","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"jax","reach":{"status":"ok","spdx":"Apache-2.0"}},{"paper_slug":"multimodal-masked-autoencoders-learn","repo_url":"https://github.com/KAIST-AILab/MaskedAutoencoder-Jax","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"jax","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"contrastive-learning","task_name":"Contrastive Learning"}],"methods":[{"method_slug":"adam","method_name":"Adam"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"attention-dropout","method_name":"Attention Dropout"},{"method_slug":"bert","method_name":"BERT"},{"method_slug":"contrastive-learning","method_name":"Contrastive Learning"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"linear-warmup-with-linear-decay","method_name":"Linear Warmup With Linear Decay"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"weight-decay","method_name":"Weight Decay"},{"method_slug":"wordpiece","method_name":"WordPiece"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2205.14204","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2205.14204"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/young-geng/m3ae_public","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/snoop2head/ehrxqa-2024","reach":{"status":"ok","spdx":"Apache-2.0"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/KAIST-AILab/MaskedAutoencoder-Jax","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran_draft_wrong":3,"unverified":9},"by_repo_kind":{"official":{"samples":3,"ran":3,"repositories":1},"listed":{"samples":9,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"4cdd517dc71ee948","entry":"create_train_step","repo":"young-geng/m3ae_public","repo_kind":"official","path":"m3ae/m3ae_main.py","file_url":"https://github.com/young-geng/m3ae_public/blob/HEAD/m3ae/m3ae_main.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"4cdd517dc71ee948"}},{"code_sha256_prefix":"cd404f2cdd3539c0","entry":"create_train_step","repo":"young-geng/m3ae_public","repo_kind":"official","path":"m3ae/finetune_main.py","file_url":"https://github.com/young-geng/m3ae_public/blob/HEAD/m3ae/finetune_main.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"cd404f2cdd3539c0"}},{"code_sha256_prefix":"c59ac510d14ebe8e","entry":"create_train_step","repo":"young-geng/m3ae_public","repo_kind":"official","path":"m3ae/linear_main.py","file_url":"https://github.com/young-geng/m3ae_public/blob/HEAD/m3ae/linear_main.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"c59ac510d14ebe8e"}},{"code_sha256_prefix":"a0ce0e70c5f4edc1","entry":"auto_augment_factory","repo":"KAIST-AILab/MaskedAutoencoder-Jax","repo_kind":"listed","path":"src/dataset.py","file_url":"https://github.com/KAIST-AILab/MaskedAutoencoder-Jax/blob/HEAD/src/dataset.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"a0ce0e70c5f4edc1"}},{"code_sha256_prefix":"0c35ede9700b8e33","entry":"collate_and_shuffle","repo":"KAIST-AILab/MaskedAutoencoder-Jax","repo_kind":"listed","path":"src/dataset.py","file_url":"https://github.com/KAIST-AILab/MaskedAutoencoder-Jax/blob/HEAD/src/dataset.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"0c35ede9700b8e33"}},{"code_sha256_prefix":"a1d3a7c11f3b5554","entry":"create_transforms","repo":"KAIST-AILab/MaskedAutoencoder-Jax","repo_kind":"listed","path":"src/dataset.py","file_url":"https://github.com/KAIST-AILab/MaskedAutoencoder-Jax/blob/HEAD/src/dataset.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"a1d3a7c11f3b5554"}},{"code_sha256_prefix":"122956528203dc19","entry":"fixed_sincos2d_embeddings","repo":"KAIST-AILab/MaskedAutoencoder-Jax","repo_kind":"listed","path":"src/utils.py","file_url":"https://github.com/KAIST-AILab/MaskedAutoencoder-Jax/blob/HEAD/src/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"122956528203dc19"}},{"code_sha256_prefix":"81e81f440f6126dd","entry":"get_layer_index_fn","repo":"KAIST-AILab/MaskedAutoencoder-Jax","repo_kind":"listed","path":"src/utils.py","file_url":"https://github.com/KAIST-AILab/MaskedAutoencoder-Jax/blob/HEAD/src/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"81e81f440f6126dd"}},{"code_sha256_prefix":"cd63725ed150c8be","entry":"mask_intersection","repo":"KAIST-AILab/MaskedAutoencoder-Jax","repo_kind":"listed","path":"src/utils_mae.py","file_url":"https://github.com/KAIST-AILab/MaskedAutoencoder-Jax/blob/HEAD/src/utils_mae.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"cd63725ed150c8be"}},{"code_sha256_prefix":"e0f860b8706ea27b","entry":"mask_not","repo":"KAIST-AILab/MaskedAutoencoder-Jax","repo_kind":"listed","path":"src/utils_mae.py","file_url":"https://github.com/KAIST-AILab/MaskedAutoencoder-Jax/blob/HEAD/src/utils_mae.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"e0f860b8706ea27b"}},{"code_sha256_prefix":"d049b3e2054cb151","entry":"mask_union","repo":"KAIST-AILab/MaskedAutoencoder-Jax","repo_kind":"listed","path":"src/utils_mae.py","file_url":"https://github.com/KAIST-AILab/MaskedAutoencoder-Jax/blob/HEAD/src/utils_mae.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"d049b3e2054cb151"}},{"code_sha256_prefix":"75ee7d495c2657a4","entry":"modified_lamb","repo":"KAIST-AILab/MaskedAutoencoder-Jax","repo_kind":"listed","path":"src/utils.py","file_url":"https://github.com/KAIST-AILab/MaskedAutoencoder-Jax/blob/HEAD/src/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"75ee7d495c2657a4"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}