{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/4m-massively-multimodal-masked-modeling-1","title":"4M: Massively Multimodal Masked Modeling","arxiv_id":"2312.06647","date":"2023-12-11","proceeding":"NeurIPS 2023 11","authors":["David Mizrahi","Roman Bachmann","Oğuzhan Fatih Kar","Teresa Yeo","Mingfei Gao","Afshin Dehghan","Amir Zamir"],"abstract":"Current machine learning models for vision are often highly specialized and limited to a single modality and task. In contrast, recent large language models exhibit a wide range of capabilities, hinting at a possibility for similarly versatile models in computer vision. In this paper, we take a step in this direction and propose a multimodal training scheme called 4M. It consists of training a single unified Transformer encoder-decoder using a masked modeling objective across a wide range of input/output modalities - including text, images, geometric, and semantic modalities, as well as neural network feature maps. 4M achieves scalability by unifying the representation space of all modalities through mapping them into discrete tokens and performing multimodal masked modeling on a small randomized subset of tokens. 4M leads to models that exhibit several key capabilities: (1) they can perform a diverse set of vision tasks out of the box, (2) they excel when fine-tuned for unseen downstream tasks or new input modalities, and (3) they can function as a generative model that can be conditioned on arbitrary modalities, enabling a wide variety of expressive multimodal editing capabilities with remarkable flexibility. Through experimental analyses, we demonstrate the potential of 4M for training versatile and scalable foundation models for vision tasks, setting the stage for further exploration in multimodal learning for vision and other domains.","url_abs":"https://arxiv.org/abs/2312.06647v1","url_pdf":"https://arxiv.org/pdf/2312.06647v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"4m-massively-multimodal-masked-modeling-1","repo_url":"https://github.com/apple/ml-4m","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"decoder","task_name":"Decoder"}],"methods":[{"method_slug":"absolute-position-encodings","method_name":"Absolute Position Encodings"},{"method_slug":"adam","method_name":"Adam"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"bpe","method_name":"BPE"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"label-smoothing","method_name":"Label Smoothing"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"position-wise-feed-forward-layer","method_name":"Position-Wise Feed-Forward Layer"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"set","method_name":"SET"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"transformer","method_name":"Transformer"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2312.06647","atlas_url":"https://app.syntology.ai/?focus=2312.06647","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2312.06647"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/apple/ml-4m","reach":null}],"summary":{"ran":8,"ran_fixture":1,"ran_honours":2,"unverified":3},"by_repo_kind":{"official":{"samples":14,"ran":11,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"469d6eab97788ae3","entry":"Attention","repo":"apple/ml-4m","repo_kind":"official","path":"fourm/models/fm.py","file_url":"https://github.com/apple/ml-4m/blob/HEAD/fourm/models/fm.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"469d6eab97788ae3"}},{"code_sha256_prefix":"92c05b09aa8b432d","entry":"CrossAttention","repo":"apple/ml-4m","repo_kind":"official","path":"fourm/models/fm.py","file_url":"https://github.com/apple/ml-4m/blob/HEAD/fourm/models/fm.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"92c05b09aa8b432d"}},{"code_sha256_prefix":"0e191e0cd5b78fae","entry":"DropPath","repo":"apple/ml-4m","repo_kind":"official","path":"fourm/models/fm.py","file_url":"https://github.com/apple/ml-4m/blob/HEAD/fourm/models/fm.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"0e191e0cd5b78fae"}},{"code_sha256_prefix":"80a8c68c90e49910","entry":"GatedMlp","repo":"apple/ml-4m","repo_kind":"official","path":"fourm/models/fm.py","file_url":"https://github.com/apple/ml-4m/blob/HEAD/fourm/models/fm.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"80a8c68c90e49910"}},{"code_sha256_prefix":"aa04da21f78ede23","entry":"LayerNorm","repo":"apple/ml-4m","repo_kind":"official","path":"fourm/models/fm.py","file_url":"https://github.com/apple/ml-4m/blob/HEAD/fourm/models/fm.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"aa04da21f78ede23"}},{"code_sha256_prefix":"fb956c775e434a63","entry":"Mlp","repo":"apple/ml-4m","repo_kind":"official","path":"fourm/models/fm.py","file_url":"https://github.com/apple/ml-4m/blob/HEAD/fourm/models/fm.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"fb956c775e434a63"}},{"code_sha256_prefix":"8e46b7a14afdc415","entry":"NormAttention","repo":"apple/ml-4m","repo_kind":"official","path":"fourm/models/fm.py","file_url":"https://github.com/apple/ml-4m/blob/HEAD/fourm/models/fm.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"8e46b7a14afdc415"}},{"code_sha256_prefix":"be370600261ed57e","entry":"NormCrossAttention","repo":"apple/ml-4m","repo_kind":"official","path":"fourm/models/fm.py","file_url":"https://github.com/apple/ml-4m/blob/HEAD/fourm/models/fm.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"be370600261ed57e"}},{"code_sha256_prefix":"e60d6e23233f6f4b","entry":"drop_path","repo":"apple/ml-4m","repo_kind":"official","path":"fourm/models/fm.py","file_url":"https://github.com/apple/ml-4m/blob/HEAD/fourm/models/fm.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"e60d6e23233f6f4b"}},{"code_sha256_prefix":"1c442a156b80c649","entry":"get_value","repo":"apple/ml-4m","repo_kind":"official","path":"fourm/demo_4M_sampler.py","file_url":"https://github.com/apple/ml-4m/blob/HEAD/fourm/demo_4M_sampler.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"1c442a156b80c649"}},{"code_sha256_prefix":"3369db8cc5790d44","entry":"softmax1","repo":"apple/ml-4m","repo_kind":"official","path":"fourm/models/fm.py","file_url":"https://github.com/apple/ml-4m/blob/HEAD/fourm/models/fm.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"3369db8cc5790d44"}},{"code_sha256_prefix":"66f58c21558cf0b3","entry":"Block","repo":"apple/ml-4m","repo_kind":"official","path":"fourm/models/fm.py","file_url":"https://github.com/apple/ml-4m/blob/HEAD/fourm/models/fm.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"66f58c21558cf0b3"}},{"code_sha256_prefix":"61b6a9e0db1b82ed","entry":"DecoderBlock","repo":"apple/ml-4m","repo_kind":"official","path":"fourm/models/fm.py","file_url":"https://github.com/apple/ml-4m/blob/HEAD/fourm/models/fm.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"61b6a9e0db1b82ed"}},{"code_sha256_prefix":"f975543590a1bbd0","entry":"FourM","repo":"apple/ml-4m","repo_kind":"official","path":"fourm/models/fm.py","file_url":"https://github.com/apple/ml-4m/blob/HEAD/fourm/models/fm.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"f975543590a1bbd0"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}