{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/multilingual-multimodal-learning-with-machine","title":"Multilingual Multimodal Learning with Machine Translated Text","arxiv_id":"2210.13134","date":"2022-10-24","proceeding":null,"authors":["Chen Qiu","Dan Oneata","Emanuele Bugliarello","Stella Frank","Desmond Elliott"],"abstract":"Most vision-and-language pretraining research focuses on English tasks. However, the creation of multilingual multimodal evaluation datasets (e.g. Multi30K, xGQA, XVNLI, and MaRVL) poses a new challenge in finding high-quality training data that is both multilingual and multimodal. In this paper, we investigate whether machine translating English multimodal data can be an effective proxy for the lack of readily available multilingual data. We call this framework TD-MML: Translated Data for Multilingual Multimodal Learning, and it can be applied to any multimodal dataset and model. We apply it to both pretraining and fine-tuning data with a state-of-the-art model. In order to prevent models from learning from low-quality translated text, we propose two metrics for automatically removing such translations from the resulting datasets. In experiments on five tasks across 20 languages in the IGLUE benchmark, we show that translated data can provide a useful signal for multilingual multimodal learning, both at pretraining and fine-tuning.","url_abs":"https://arxiv.org/abs/2210.13134v1","url_pdf":"https://arxiv.org/pdf/2210.13134v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"multilingual-multimodal-learning-with-machine","repo_url":"https://github.com/danoneata/td-mml","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"zero-shot-cross-lingual-image-to-text","task_name":"Zero-Shot Cross-Lingual Image-to-Text Retrieval"},{"task_slug":"zero-shot-cross-lingual-text-to-image","task_name":"Zero-Shot Cross-Lingual Text-to-Image Retrieval"},{"task_slug":"zero-shot-cross-lingual-visual-natural","task_name":"Zero-Shot Cross-Lingual Visual Natural Language Inference"},{"task_slug":"zero-shot-cross-lingual-visual-question","task_name":"Zero-Shot Cross-Lingual Visual Question Answering"},{"task_slug":"zero-shot-cross-lingual-visual-reasoning","task_name":"Zero-Shot Cross-Lingual Visual Reasoning"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2210.13134","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2210.13134"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/danoneata/td-mml","reach":null}],"summary":{"unverified":1},"by_repo_kind":{"official":{"samples":1,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"419f476e0d483647","entry":"build_roi_box_head","repo":"danoneata/td-mml","repo_kind":"official","path":"features_extraction/mmf/github/vqa-maskrcnn-benchmark/maskrcnn_benchmark/modeling/roi_heads/box_head/box_head.py","file_url":"https://github.com/danoneata/td-mml/blob/HEAD/features_extraction/mmf/github/vqa-maskrcnn-benchmark/maskrcnn_benchmark/modeling/roi_heads/box_head/box_head.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"419f476e0d483647"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}