{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/cliptrans-transferring-visual-knowledge-with","title":"CLIPTrans: Transferring Visual Knowledge with Pre-trained Models for Multimodal Machine Translation","arxiv_id":"2308.15226","date":"2023-08-29","proceeding":"ICCV 2023 1","authors":["Devaansh Gupta","Siddhant Kharbanda","Jiawei Zhou","Wanhua Li","Hanspeter Pfister","Donglai Wei"],"abstract":"There has been a growing interest in developing multimodal machine translation (MMT) systems that enhance neural machine translation (NMT) with visual knowledge. This problem setup involves using images as auxiliary information during training, and more recently, eliminating their use during inference. Towards this end, previous works face a challenge in training powerful MMT models from scratch due to the scarcity of annotated multilingual vision-language data, especially for low-resource languages. Simultaneously, there has been an influx of multilingual pre-trained models for NMT and multimodal pre-trained models for vision-language tasks, primarily in English, which have shown exceptional generalisation ability. However, these are not directly applicable to MMT since they do not provide aligned multimodal multilingual features for generative tasks. To alleviate this issue, instead of designing complex modules for MMT, we propose CLIPTrans, which simply adapts the independently pre-trained multimodal M-CLIP and the multilingual mBART. In order to align their embedding spaces, mBART is conditioned on the M-CLIP features by a prefix sequence generated through a lightweight mapping network. We train this in a two-stage pipeline which warms up the model with image captioning before the actual translation task. Through experiments, we demonstrate the merits of this framework and consequently push forward the state-of-the-art across standard benchmarks by an average of +2.67 BLEU. The code can be found at www.github.com/devaansh100/CLIPTrans.","url_abs":"https://arxiv.org/abs/2308.15226v1","url_pdf":"https://arxiv.org/pdf/2308.15226v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"cliptrans-transferring-visual-knowledge-with","repo_url":"https://github.com/devaansh100/cliptrans","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":"image-captioning","task_name":"Image Captioning"},{"task_slug":"machine-translation","task_name":"Machine Translation"},{"task_slug":"multimodal-machine-translation","task_name":"Multimodal Machine Translation"},{"task_slug":"nmt","task_name":"NMT"},{"task_slug":"translation","task_name":"Translation"}],"methods":[{"method_slug":"align","method_name":"ALIGN"},{"method_slug":"mbart","method_name":"mBART"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2308.15226","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2308.15226"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/devaansh100/cliptrans","reach":{"status":"ok"}},{"provenance":"deterministic:regex_extraction","url":"https://github.com/devaansh100/CLIPTrans","reach":{"status":"ok"}}],"summary":{"ran":5,"unverified":2},"by_repo_kind":{"official":{"samples":7,"ran":5,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":7,"samples":[{"code_sha256_prefix":"4e0fe83961359644","entry":"collate_images","repo":"devaansh100/cliptrans","repo_kind":"official","path":"src/collate_fns.py","file_url":"https://github.com/devaansh100/cliptrans/blob/HEAD/src/collate_fns.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"4e0fe83961359644"}},{"code_sha256_prefix":"1cd1e0a0b299e66b","entry":"collate_texts","repo":"devaansh100/cliptrans","repo_kind":"official","path":"src/collate_fns.py","file_url":"https://github.com/devaansh100/cliptrans/blob/HEAD/src/collate_fns.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"1cd1e0a0b299e66b"}},{"code_sha256_prefix":"e28486c35f06488a","entry":"get_lang_code","repo":"devaansh100/cliptrans","repo_kind":"official","path":"src/utils/model_utils.py","file_url":"https://github.com/devaansh100/cliptrans/blob/HEAD/src/utils/model_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"e28486c35f06488a"}},{"code_sha256_prefix":"583071847929d508","entry":"read_image","repo":"devaansh100/cliptrans","repo_kind":"official","path":"src/utils/data_utils.py","file_url":"https://github.com/devaansh100/cliptrans/blob/HEAD/src/utils/data_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"583071847929d508"}},{"code_sha256_prefix":"c7a33917dd745148","entry":"send_to_cuda","repo":"devaansh100/cliptrans","repo_kind":"official","path":"src/utils/model_utils.py","file_url":"https://github.com/devaansh100/cliptrans/blob/HEAD/src/utils/model_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"c7a33917dd745148"}},{"code_sha256_prefix":"e59c6d51dd3affdf","entry":"CLIPTrans","repo":"devaansh100/CLIPTrans","repo_kind":"official","path":"src/models/model.py","file_url":"https://github.com/devaansh100/CLIPTrans/blob/HEAD/src/models/model.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"e59c6d51dd3affdf"}},{"code_sha256_prefix":"3ce3bd57f7a96062","entry":"collate_multi","repo":"devaansh100/cliptrans","repo_kind":"official","path":"src/collate_fns.py","file_url":"https://github.com/devaansh100/cliptrans/blob/HEAD/src/collate_fns.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"3ce3bd57f7a96062"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}