{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/mammut-a-simple-architecture-for-joint","title":"MaMMUT: A Simple Architecture for Joint Learning for MultiModal Tasks","arxiv_id":"2303.16839","date":"2023-03-29","proceeding":null,"authors":["Weicheng Kuo","AJ Piergiovanni","Dahun Kim","Xiyang Luo","Ben Caine","Wei Li","Abhijit Ogale","Luowei Zhou","Andrew Dai","Zhifeng Chen","Claire Cui","Anelia Angelova"],"abstract":"The development of language models have moved from encoder-decoder to decoder-only designs. In addition, we observe that the two most popular multimodal tasks, the generative and contrastive tasks, are nontrivial to accommodate in one architecture, and further need adaptations for downstream tasks. We propose a novel paradigm of training with a decoder-only model for multimodal tasks, which is surprisingly effective in jointly learning of these disparate vision-language tasks. This is done with a simple model, called MaMMUT. It consists of a single vision encoder and a text decoder, and is able to accommodate contrastive and generative learning by a novel two-pass approach on the text decoder. We demonstrate that joint learning of these diverse objectives is simple, effective, and maximizes the weight-sharing of the model across these tasks. Furthermore, the same architecture enables straightforward extensions to open-vocabulary object detection and video-language tasks. The model tackles a diverse range of tasks, while being modest in capacity. Our model achieves the state of the art on image-text and text-image retrieval, video question answering and open-vocabulary detection tasks, outperforming much larger and more extensively trained foundational models. It shows very competitive results on VQA and Video Captioning, especially considering its capacity. Ablations confirm the flexibility and advantages of our approach.","url_abs":"https://arxiv.org/abs/2303.16839v3","url_pdf":"https://arxiv.org/pdf/2303.16839v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"mammut-a-simple-architecture-for-joint","repo_url":"https://github.com/lucidrains/mammut-pytorch","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"cross-modal-retrieval","task_name":"Cross-Modal Retrieval"},{"task_slug":"decoder","task_name":"Decoder"},{"task_slug":"image-retrieval","task_name":"Image Retrieval"},{"task_slug":"object-detection","task_name":"Object Detection"},{"task_slug":"open-vocabulary-object-detection","task_name":"Open Vocabulary Object Detection"},{"task_slug":"open-vocabulary-object-detection","task_name":"Open-vocabulary object detection"},{"task_slug":"question-answering","task_name":"Question Answering"},{"task_slug":"retrieval","task_name":"Retrieval"},{"task_slug":"video-captioning","task_name":"Video Captioning"},{"task_slug":"video-question-answering","task_name":"Video Question Answering"},{"task_slug":"visual-question-answering-1","task_name":"Visual Question Answering"},{"task_slug":"visual-question-answering","task_name":"Visual Question Answering (VQA)"},{"task_slug":"object-detection-1","task_name":"object-detection"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/cross-modal-retrieval-on-coco-2014","task":"Cross-Modal Retrieval","dataset":"COCO 2014","model":"MaMMUT (ours)","rank_in_archive_order":36,"of":36,"metrics":{"Image-to-text R@1":"70.7","Image-to-text R@10":"93.7","Image-to-text R@5":"89.1"},"uses_additional_data":false},{"leaderboard":"/sota/image-retrieval-on-flickr30k","task":"Image Retrieval","dataset":"Flickr30k","model":"MaMMUT (ours)","rank_in_archive_order":4,"of":9,"metrics":{"Image-to-text R@1":"94.9","Image-to-text R@10":"99.9","Image-to-text R@5":"99.5","Recall@1":"82.5","Recall@10":"98","Recall@5":"96"},"uses_additional_data":false},{"leaderboard":"/sota/question-answering-on-coco-visual-question","task":"Question Answering","dataset":"COCO Visual Question Answering (VQA) real images 1.0 open ended","model":"MaMMUT (2B)","rank_in_archive_order":1,"of":1,"metrics":{"Test":"80.8"},"uses_additional_data":false},{"leaderboard":"/sota/video-captioning-on-msr-vtt-1","task":"Video Captioning","dataset":"MSR-VTT","model":"MaMMUT (ours)","rank_in_archive_order":7,"of":24,"metrics":{"CIDEr":"73.6"},"uses_additional_data":false},{"leaderboard":"/sota/video-captioning-on-msvd-1","task":"Video Captioning","dataset":"MSVD","model":"MaMMUT","rank_in_archive_order":1,"of":16,"metrics":{"CIDEr":"195.6"},"uses_additional_data":false},{"leaderboard":"/sota/visual-question-answering-on-coco-visual-5","task":"Visual Question Answering","dataset":"COCO Visual Question Answering (VQA) real images 2.0 open ended","model":"MaMMUT (2B)","rank_in_archive_order":1,"of":1,"metrics":{"Percentage correct":"80.7"},"uses_additional_data":false},{"leaderboard":"/sota/visual-question-answering-on-msrvtt-qa-1","task":"Visual Question Answering (VQA)","dataset":"MSRVTT-QA","model":"MaMMUT","rank_in_archive_order":2,"of":34,"metrics":{"Accuracy":"0.495"},"uses_additional_data":true},{"leaderboard":"/sota/visual-question-answering-on-msvd-qa-1","task":"Visual Question Answering (VQA)","dataset":"MSVD-QA","model":"MaMMUT (ours)","rank_in_archive_order":3,"of":36,"metrics":{"Accuracy":".602"},"uses_additional_data":true}],"syntology":{"syntology_url":"https://syntology.ai/paper/2303.16839","atlas_url":"https://app.syntology.ai/?focus=2303.16839","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2303.16839"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/lucidrains/mammut-pytorch","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran_violates":3},"by_repo_kind":{"listed":{"samples":3,"ran":3,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"856d86b39c6005b4","entry":"divisible_by","repo":"lucidrains/mammut-pytorch","repo_kind":"listed","path":"mammut_pytorch/mammut_pytorch.py","file_url":"https://github.com/lucidrains/mammut-pytorch/blob/HEAD/mammut_pytorch/mammut_pytorch.py","link_basis":"harvester_set","language":"python","status":"ran_violates","verification_level":2,"contract_check":"VIOLATES","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"856d86b39c6005b4"}},{"code_sha256_prefix":"60fff7c3c400d7ff","entry":"default","repo":"lucidrains/mammut-pytorch","repo_kind":"listed","path":"mammut_pytorch/mammut_pytorch.py","file_url":"https://github.com/lucidrains/mammut-pytorch/blob/HEAD/mammut_pytorch/mammut_pytorch.py","link_basis":"harvester_set","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"well_formed","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"60fff7c3c400d7ff"}},{"code_sha256_prefix":"aa5486a3650902d8","entry":"exists","repo":"lucidrains/mammut-pytorch","repo_kind":"listed","path":"mammut_pytorch/mammut_pytorch.py","file_url":"https://github.com/lucidrains/mammut-pytorch/blob/HEAD/mammut_pytorch/mammut_pytorch.py","link_basis":"harvester_set","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"aa5486a3650902d8"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}