{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/mitigating-modality-collapse-in-multimodal","title":"Mitigating Modality Collapse in Multimodal VAEs via Impartial Optimization","arxiv_id":"2206.04496","date":"2022-06-09","proceeding":null,"authors":["Adrián Javaloy","Maryam Meghdadi","Isabel Valera"],"abstract":"A number of variational autoencoders (VAEs) have recently emerged with the aim of modeling multimodal data, e.g., to jointly model images and their corresponding captions. Still, multimodal VAEs tend to focus solely on a subset of the modalities, e.g., by fitting the image while neglecting the caption. We refer to this limitation as modality collapse. In this work, we argue that this effect is a consequence of conflicting gradients during multimodal VAE training. We show how to detect the sub-graphs in the computational graphs where gradients conflict (impartiality blocks), as well as how to leverage existing gradient-conflict solutions from multitask learning to mitigate modality collapse. That is, to ensure impartial optimization across modalities. We apply our training framework to several multimodal VAE models, losses and datasets from the literature, and empirically show that our framework significantly improves the reconstruction performance, conditional generation, and coherence of the latent space across modalities.","url_abs":"https://arxiv.org/abs/2206.04496v1","url_pdf":"https://arxiv.org/pdf/2206.04496v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"mitigating-modality-collapse-in-multimodal","repo_url":"https://github.com/adrianjav/impartial-vaes","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"pytorch","reach":null}],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2206.04496","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2206.04496"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/adrianjav/impartial-vaes","reach":null}],"summary":{"ran":1,"ran_draft_wrong":4,"unverified":1},"by_repo_kind":{"official":{"samples":6,"ran":5,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"52138dfae15b9b1c","entry":"IMTLG","repo":"adrianjav/impartial-vaes","repo_kind":"official","path":"heterogeneous/torchmoo/mtl.py","file_url":"https://github.com/adrianjav/impartial-vaes/blob/HEAD/heterogeneous/torchmoo/mtl.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"52138dfae15b9b1c"}},{"code_sha256_prefix":"ca6dfaa6e9d166c6","entry":"batch_product","repo":"adrianjav/impartial-vaes","repo_kind":"official","path":"heterogeneous/torchmoo/mtl.py","file_url":"https://github.com/adrianjav/impartial-vaes/blob/HEAD/heterogeneous/torchmoo/mtl.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ca6dfaa6e9d166c6"}},{"code_sha256_prefix":"d68a00f830d4ed18","entry":"divide","repo":"adrianjav/impartial-vaes","repo_kind":"official","path":"heterogeneous/torchmoo/mtl.py","file_url":"https://github.com/adrianjav/impartial-vaes/blob/HEAD/heterogeneous/torchmoo/mtl.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d68a00f830d4ed18"}},{"code_sha256_prefix":"f5594684df370bd8","entry":"norm","repo":"adrianjav/impartial-vaes","repo_kind":"official","path":"heterogeneous/torchmoo/mtl.py","file_url":"https://github.com/adrianjav/impartial-vaes/blob/HEAD/heterogeneous/torchmoo/mtl.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f5594684df370bd8"}},{"code_sha256_prefix":"99cc145a1deff170","entry":"unitary","repo":"adrianjav/impartial-vaes","repo_kind":"official","path":"heterogeneous/torchmoo/mtl.py","file_url":"https://github.com/adrianjav/impartial-vaes/blob/HEAD/heterogeneous/torchmoo/mtl.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"99cc145a1deff170"}},{"code_sha256_prefix":"4b8b4af3c3602a0c","entry":"MOOMethod","repo":"adrianjav/impartial-vaes","repo_kind":"official","path":"heterogeneous/torchmoo/mtl.py","file_url":"https://github.com/adrianjav/impartial-vaes/blob/HEAD/heterogeneous/torchmoo/mtl.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"4b8b4af3c3602a0c"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}