{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/magvlt-masked-generative-vision-and-language","title":"MAGVLT: Masked Generative Vision-and-Language Transformer","arxiv_id":"2303.12208","date":"2023-03-21","proceeding":"CVPR 2023 1","authors":["Sungwoong Kim","DaeJin Jo","Donghoon Lee","Jongmin Kim"],"abstract":"While generative modeling on multimodal image-text data has been actively developed with large-scale paired datasets, there have been limited attempts to generate both image and text data by a single model rather than a generation of one fixed modality conditioned on the other modality. In this paper, we explore a unified generative vision-and-language (VL) model that can produce both images and text sequences. Especially, we propose a generative VL transformer based on the non-autoregressive mask prediction, named MAGVLT, and compare it with an autoregressive generative VL transformer (ARGVLT). In comparison to ARGVLT, the proposed MAGVLT enables bidirectional context encoding, fast decoding by parallel token predictions in an iterative refinement, and extended editing capabilities such as image and text infilling. For rigorous training of our MAGVLT with image-text pairs from scratch, we combine the image-to-text, text-to-image, and joint image-and-text mask prediction tasks. Moreover, we devise two additional tasks based on the step-unrolled mask prediction and the selective prediction on the mixture of two image-text pairs. Experimental results on various downstream generation tasks of VL benchmarks show that our MAGVLT outperforms ARGVLT by a large margin even with significant inference speedup. Particularly, MAGVLT achieves competitive results on both zero-shot image-to-text and text-to-image generation tasks from MS-COCO by one moderate-sized model (fewer than 500M parameters) even without the use of monomodal data and networks.","url_abs":"https://arxiv.org/abs/2303.12208v1","url_pdf":"https://arxiv.org/pdf/2303.12208v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"magvlt-masked-generative-vision-and-language","repo_url":"https://github.com/kakaobrain/magvlt","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"image-captioning","task_name":"Image Captioning"},{"task_slug":"image-generation","task_name":"Image Generation"},{"task_slug":"image-to-text","task_name":"Image to text"},{"task_slug":"prediction","task_name":"Prediction"},{"task_slug":"text-infilling","task_name":"Text Infilling"},{"task_slug":"text-to-image-generation-1","task_name":"Text to Image Generation"},{"task_slug":"text-to-image-generation","task_name":"Text-to-Image Generation"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":null,"atlas_url":"https://app.syntology.ai/?focus=2303.12208","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2303.12208"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/kakaobrain/magvlt","reach":null}],"summary":{"ran":7,"unverified":3},"by_repo_kind":{"official":{"samples":10,"ran":7,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"89844ef2ac1027a2","entry":"Block","repo":"kakaobrain/magvlt","repo_kind":"official","path":"magvlt/models/stage2/transformers.py","file_url":"https://github.com/kakaobrain/magvlt/blob/HEAD/magvlt/models/stage2/transformers.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"89844ef2ac1027a2"}},{"code_sha256_prefix":"0b3c58ba91896ec0","entry":"GELU","repo":"kakaobrain/magvlt","repo_kind":"official","path":"magvlt/models/stage2/transformers.py","file_url":"https://github.com/kakaobrain/magvlt/blob/HEAD/magvlt/models/stage2/transformers.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"0b3c58ba91896ec0"}},{"code_sha256_prefix":"1ae0447b138f7394","entry":"LMLayer","repo":"kakaobrain/magvlt","repo_kind":"official","path":"magvlt/models/stage2/transformers.py","file_url":"https://github.com/kakaobrain/magvlt/blob/HEAD/magvlt/models/stage2/transformers.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"1ae0447b138f7394"}},{"code_sha256_prefix":"2f73823d89add892","entry":"MultiHeadSelfAttention","repo":"kakaobrain/magvlt","repo_kind":"official","path":"magvlt/models/stage2/transformers.py","file_url":"https://github.com/kakaobrain/magvlt/blob/HEAD/magvlt/models/stage2/transformers.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"2f73823d89add892"}},{"code_sha256_prefix":"08e6f64d649cf0e9","entry":"PositionalEmbed","repo":"kakaobrain/magvlt","repo_kind":"official","path":"magvlt/models/stage2/transformers.py","file_url":"https://github.com/kakaobrain/magvlt/blob/HEAD/magvlt/models/stage2/transformers.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"08e6f64d649cf0e9"}},{"code_sha256_prefix":"363e79d3badd3e32","entry":"Sinusoidal1DPositionalEmbed","repo":"kakaobrain/magvlt","repo_kind":"official","path":"magvlt/models/stage2/transformers.py","file_url":"https://github.com/kakaobrain/magvlt/blob/HEAD/magvlt/models/stage2/transformers.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"363e79d3badd3e32"}},{"code_sha256_prefix":"1bad1f729cc2768c","entry":"Sinusoidal2DPositionalEmbed","repo":"kakaobrain/magvlt","repo_kind":"official","path":"magvlt/models/stage2/transformers.py","file_url":"https://github.com/kakaobrain/magvlt/blob/HEAD/magvlt/models/stage2/transformers.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"1bad1f729cc2768c"}},{"code_sha256_prefix":"94eaf59881a7f6f3","entry":"ModelTemplate","repo":"kakaobrain/magvlt","repo_kind":"official","path":"magvlt/models/stage2/transformers.py","file_url":"https://github.com/kakaobrain/magvlt/blob/HEAD/magvlt/models/stage2/transformers.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"94eaf59881a7f6f3"}},{"code_sha256_prefix":"53ebad6e00fab368","entry":"SPECIAL_TOKENS","repo":"kakaobrain/magvlt","repo_kind":"official","path":"magvlt/models/stage2/transformers.py","file_url":"https://github.com/kakaobrain/magvlt/blob/HEAD/magvlt/models/stage2/transformers.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"53ebad6e00fab368"}},{"code_sha256_prefix":"6edc6c8c22bba74b","entry":"Transformer1d","repo":"kakaobrain/magvlt","repo_kind":"official","path":"magvlt/models/stage2/transformers.py","file_url":"https://github.com/kakaobrain/magvlt/blob/HEAD/magvlt/models/stage2/transformers.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"6edc6c8c22bba74b"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}