{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/learning-multimodal-data-augmentation-in","title":"Learning Multimodal Data Augmentation in Feature Space","arxiv_id":"2212.14453","date":"2022-12-29","proceeding":null,"authors":["Zichang Liu","Zhiqiang Tang","Xingjian Shi","Aston Zhang","Mu Li","Anshumali Shrivastava","Andrew Gordon Wilson"],"abstract":"The ability to jointly learn from multiple modalities, such as text, audio, and visual data, is a defining feature of intelligent systems. While there have been promising advances in designing neural networks to harness multimodal data, the enormous success of data augmentation currently remains limited to single-modality tasks like image classification. Indeed, it is particularly difficult to augment each modality while preserving the overall semantic structure of the data; for example, a caption may no longer be a good description of an image after standard augmentations have been applied, such as translation. Moreover, it is challenging to specify reasonable transformations that are not tailored to a particular modality. In this paper, we introduce LeMDA, Learning Multimodal Data Augmentation, an easy-to-use method that automatically learns to jointly augment multimodal data in feature space, with no constraints on the identities of the modalities or the relationship between modalities. We show that LeMDA can (1) profoundly improve the performance of multimodal deep learning architectures, (2) apply to combinations of modalities that have not been previously considered, and (3) achieve state-of-the-art results on a wide range of applications comprised of image, text, and tabular data.","url_abs":"https://arxiv.org/abs/2212.14453v2","url_pdf":"https://arxiv.org/pdf/2212.14453v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"learning-multimodal-data-augmentation-in","repo_url":"https://github.com/lzcemma/lemda","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"data-augmentation","task_name":"Data Augmentation"},{"task_slug":"image-classification","task_name":"Image Classification"},{"task_slug":"multimodal-deep-learning","task_name":"Multimodal Deep Learning"},{"task_slug":"image-classification","task_name":"image-classification"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2212.14453","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2212.14453"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/lzcemma/LeMDA","reach":{"status":"ok","spdx":"Apache-2.0"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/lzcemma/lemda","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran":1,"ran_violates":2,"ran_draft_wrong":6,"unverified":1},"by_repo_kind":{"official":{"samples":10,"ran":9,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"c3a5691752afb687","entry":"FeatureMetadata","repo":"lzcemma/lemda","repo_kind":"official","path":"core/src/autogluon/core/augmentation/distill_utils.py","file_url":"https://github.com/lzcemma/lemda/blob/HEAD/core/src/autogluon/core/augmentation/distill_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"c3a5691752afb687"}},{"code_sha256_prefix":"d12c187041543caf","entry":"check_if_datetime_as_object_feature","repo":"lzcemma/lemda","repo_kind":"official","path":"core/src/autogluon/core/augmentation/distill_utils.py","file_url":"https://github.com/lzcemma/lemda/blob/HEAD/core/src/autogluon/core/augmentation/distill_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"d12c187041543caf"}},{"code_sha256_prefix":"7d2c61e10c7dedda","entry":"check_if_nlp_feature","repo":"lzcemma/lemda","repo_kind":"official","path":"core/src/autogluon/core/augmentation/distill_utils.py","file_url":"https://github.com/lzcemma/lemda/blob/HEAD/core/src/autogluon/core/augmentation/distill_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"7d2c61e10c7dedda"}},{"code_sha256_prefix":"dc8b9153485cadb8","entry":"get_type_family_raw","repo":"lzcemma/lemda","repo_kind":"official","path":"core/src/autogluon/core/augmentation/distill_utils.py","file_url":"https://github.com/lzcemma/lemda/blob/HEAD/core/src/autogluon/core/augmentation/distill_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"dc8b9153485cadb8"}},{"code_sha256_prefix":"a48376b2d4e3af99","entry":"get_type_group_map","repo":"lzcemma/lemda","repo_kind":"official","path":"core/src/autogluon/core/augmentation/distill_utils.py","file_url":"https://github.com/lzcemma/lemda/blob/HEAD/core/src/autogluon/core/augmentation/distill_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"a48376b2d4e3af99"}},{"code_sha256_prefix":"1f5792e2cc532804","entry":"get_type_group_map_special","repo":"lzcemma/lemda","repo_kind":"official","path":"core/src/autogluon/core/augmentation/distill_utils.py","file_url":"https://github.com/lzcemma/lemda/blob/HEAD/core/src/autogluon/core/augmentation/distill_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"1f5792e2cc532804"}},{"code_sha256_prefix":"3196d80d7d4cd616","entry":"get_type_map_raw","repo":"lzcemma/lemda","repo_kind":"official","path":"core/src/autogluon/core/augmentation/distill_utils.py","file_url":"https://github.com/lzcemma/lemda/blob/HEAD/core/src/autogluon/core/augmentation/distill_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"3196d80d7d4cd616"}},{"code_sha256_prefix":"298dc9e7e784d15e","entry":"get_type_map_special","repo":"lzcemma/lemda","repo_kind":"official","path":"core/src/autogluon/core/augmentation/distill_utils.py","file_url":"https://github.com/lzcemma/lemda/blob/HEAD/core/src/autogluon/core/augmentation/distill_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"298dc9e7e784d15e"}},{"code_sha256_prefix":"0745c3b8daaa7e69","entry":"get_types_special","repo":"lzcemma/lemda","repo_kind":"official","path":"core/src/autogluon/core/augmentation/distill_utils.py","file_url":"https://github.com/lzcemma/lemda/blob/HEAD/core/src/autogluon/core/augmentation/distill_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"0745c3b8daaa7e69"}},{"code_sha256_prefix":"7d535eca508a8f22","entry":"spunge_augment","repo":"lzcemma/lemda","repo_kind":"official","path":"core/src/autogluon/core/augmentation/distill_utils.py","file_url":"https://github.com/lzcemma/lemda/blob/HEAD/core/src/autogluon/core/augmentation/distill_utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"7d535eca508a8f22"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}