{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/diffusion-models-as-masked-autoencoders","title":"Diffusion Models as Masked Autoencoders","arxiv_id":"2304.03283","date":"2023-04-06","proceeding":"ICCV 2023 1","authors":["Chen Wei","Karttikeya Mangalam","Po-Yao Huang","Yanghao Li","Haoqi Fan","Hu Xu","Huiyu Wang","Cihang Xie","Alan Yuille","Christoph Feichtenhofer"],"abstract":"There has been a longstanding belief that generation can facilitate a true understanding of visual data. In line with this, we revisit generatively pre-training visual representations in light of recent interest in denoising diffusion models. While directly pre-training with diffusion models does not produce strong representations, we condition diffusion models on masked input and formulate diffusion models as masked autoencoders (DiffMAE). Our approach is capable of (i) serving as a strong initialization for downstream recognition tasks, (ii) conducting high-quality image inpainting, and (iii) being effortlessly extended to video where it produces state-of-the-art classification accuracy. We further perform a comprehensive study on the pros and cons of design choices and build connections between diffusion models and masked autoencoders.","url_abs":"https://arxiv.org/abs/2304.03283v1","url_pdf":"https://arxiv.org/pdf/2304.03283v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"diffusion-models-as-masked-autoencoders","repo_url":"https://github.com/kimdanni/DiffMAE","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"denoising","task_name":"Denoising"},{"task_slug":"image-inpainting","task_name":"Image Inpainting"}],"methods":[{"method_slug":"diffusion","method_name":"Diffusion"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2304.03283","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2304.03283"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/kimdanni/DiffMAE","reach":null}],"summary":{"ran":7,"ran_fixture":1,"ran_honours":1,"unverified":3},"by_repo_kind":{"listed":{"samples":12,"ran":9,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":12,"samples":[{"code_sha256_prefix":"f7e0f3a36a14e867","entry":"Attention","repo":"kimdanni/DiffMAE","repo_kind":"listed","path":"models_cross.py","file_url":"https://github.com/kimdanni/DiffMAE/blob/HEAD/models_cross.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"f7e0f3a36a14e867"}},{"code_sha256_prefix":"a766b080f0eacfd2","entry":"Block","repo":"kimdanni/DiffMAE","repo_kind":"listed","path":"models_cross.py","file_url":"https://github.com/kimdanni/DiffMAE/blob/HEAD/models_cross.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"a766b080f0eacfd2"}},{"code_sha256_prefix":"563ed599da513e37","entry":"CrossAttention","repo":"kimdanni/DiffMAE","repo_kind":"listed","path":"models_cross.py","file_url":"https://github.com/kimdanni/DiffMAE/blob/HEAD/models_cross.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"563ed599da513e37"}},{"code_sha256_prefix":"edac750b243ddc24","entry":"CrossAttentionBlock","repo":"kimdanni/DiffMAE","repo_kind":"listed","path":"models_cross.py","file_url":"https://github.com/kimdanni/DiffMAE/blob/HEAD/models_cross.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"edac750b243ddc24"}},{"code_sha256_prefix":"11e42ea9da37c363","entry":"PatchEmbed","repo":"kimdanni/DiffMAE","repo_kind":"listed","path":"models_cross.py","file_url":"https://github.com/kimdanni/DiffMAE/blob/HEAD/models_cross.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"11e42ea9da37c363"}},{"code_sha256_prefix":"94ed19243846560a","entry":"TimestepEmbedder","repo":"kimdanni/DiffMAE","repo_kind":"listed","path":"models_cross.py","file_url":"https://github.com/kimdanni/DiffMAE/blob/HEAD/models_cross.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"94ed19243846560a"}},{"code_sha256_prefix":"7e8768cb8c14f240","entry":"WeightedFeatureMaps","repo":"kimdanni/DiffMAE","repo_kind":"listed","path":"models_cross.py","file_url":"https://github.com/kimdanni/DiffMAE/blob/HEAD/models_cross.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"7e8768cb8c14f240"}},{"code_sha256_prefix":"6a53acdcb72e78dd","entry":"get_1d_sincos_pos_embed_from_grid","repo":"kimdanni/DiffMAE","repo_kind":"listed","path":"models_cross.py","file_url":"https://github.com/kimdanni/DiffMAE/blob/HEAD/models_cross.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"6a53acdcb72e78dd"}},{"code_sha256_prefix":"6df5fc6a28a5d9e4","entry":"get_2d_sincos_pos_embed","repo":"kimdanni/DiffMAE","repo_kind":"listed","path":"models_cross.py","file_url":"https://github.com/kimdanni/DiffMAE/blob/HEAD/models_cross.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"6df5fc6a28a5d9e4"}},{"code_sha256_prefix":"584be512341b7ae6","entry":"CustomDDPMScheduler","repo":"kimdanni/DiffMAE","repo_kind":"listed","path":"models_cross.py","file_url":"https://github.com/kimdanni/DiffMAE/blob/HEAD/models_cross.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"584be512341b7ae6"}},{"code_sha256_prefix":"1a91090e8cbfa8d4","entry":"MaskedAutoencoderViT","repo":"kimdanni/DiffMAE","repo_kind":"listed","path":"models_cross.py","file_url":"https://github.com/kimdanni/DiffMAE/blob/HEAD/models_cross.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"1a91090e8cbfa8d4"}},{"code_sha256_prefix":"b7482295ffca3e2c","entry":"get_2d_sincos_pos_embed_from_grid","repo":"kimdanni/DiffMAE","repo_kind":"listed","path":"models_cross.py","file_url":"https://github.com/kimdanni/DiffMAE/blob/HEAD/models_cross.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"b7482295ffca3e2c"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}