Multimodal Machine Translation

Description from the archive archive 2025-07-28.

Benchmarks archive 2025-07-28

3 leaderboard tables shown for this task, 3 with rows (a “benchmark” on this site is a table with at least one row, as on /sota), ordered by row count. “Best model” is the first row in the archive's own order at snapshot; nothing is re-ranked here and metric direction is not recorded in the archive. PwC's Trend sparklines are not in the archive, so that column is omitted.

Syntology column: samples harvested from the paper's repositories and executed on synthesized fixtures; “ran” is not a correctness claim and does not order the table. A dash means no Syntology record for that paper, not a recorded non-run. Read from the graph 2026-09-24.

Libraries

Not in the archive: the export carries no per-task library table, so there is nothing to show at snapshot 2025-07-28.

Datasets archive 2025-07-28

5 datasets whose archive record lists this task, ordered by the archive's paper count.

Subtasks archive 2025-07-28

2 subtasks in the archive's task tree.

Parent tasks archive 2025-07-28

Most implemented papers archive 2025-07-28

30 shown of 38 papers with code (108 tagged with this task in all), ordered by repositories listed in the archive, not by stars (the archive holds no stars, so PwC's “Social” and “Latest” sorts cannot be reproduced). Papers without a page here are shown as plain text.

• Attention Is All You Need
  12 Jun 2017 595 repositories listed Syntology ran 600 of 946 samples · 346 unverified · 451 pointer-only (licence)
  The dominant sequence transduction models are based on complex recurrent or convolutional neural networks in an encoder-decoder configuration.
• Towards Zero-Shot Multimodal Machine Translation
  18 Jul 2024 2 repositories listed
  Current multimodal machine translation (MMT) systems rely on fully supervised data (i.
• Tackling Ambiguity with Images: Improved Multimodal Machine Translation and Contrastive Evaluation
  20 Dec 2022 2 repositories listed Syntology ran 2 of 2 samples · 0 unverified · 2 pointer-only (licence)
  One of the major challenges of machine translation (MT) is ambiguity, which can in some cases be resolved by accompanying context such as images.
• On Vision Features in Multimodal Machine Translation
  17 Mar 2022 2 repositories listed Syntology ran 2 of 3 samples · 1 unverified
  Previous work on multimodal machine translation (MMT) has focused on the way of incorporating vision features into translation but little attention is on the quality of vision models.
• Multi30K: Multilingual English-German Image Descriptions
  2 May 2016 2 repositories listed
  We introduce the Multi30K dataset to stimulate multilingual multimodal research.
• 3AM: An Ambiguity-Aware Multi-Modal Machine Translation Dataset
  29 Apr 2024 1 repository listed
  Experimental results show that MMT models trained on our dataset exhibit a greater ability to exploit visual information than those trained on other MMT datasets.
• Seamless: Multilingual Expressive and Streaming Speech Translation
  8 Dec 2023 1 repository listed
  In this work, we introduce a family of models that enable end-to-end expressive and multilingual translations in a streaming fashion.
• Video-Helpful Multimodal Machine Translation
  31 Oct 2023 1 repository listed
  In addition to the extensive training set, EVA contains a video-helpful evaluation set in which subtitles are ambiguous, and videos are guaranteed helpful for disambiguation.
• Incorporating Probing Signals into Multimodal Machine Translation via Visual Question-Answering Pairs
  26 Oct 2023 1 repository listed
  This paper presents an in-depth study of multimodal machine translation (MMT), examining the prevailing understanding that MMT systems exhibit decreased sensitivity to visual information when text inputs are complete.
• Bridging the Gap between Synthetic and Authentic Images for Multimodal Machine Translation
  20 Oct 2023 1 repository listed
  Multimodal machine translation (MMT) simultaneously takes the source sentence and a relevant image as input for translation.
• CLIPTrans: Transferring Visual Knowledge with Pre-trained Models for Multimodal Machine Translation
  29 Aug 2023 1 repository listed Syntology ran 5 of 7 samples · 2 unverified · 7 pointer-only (licence)
  Simultaneously, there has been an influx of multilingual pre-trained models for NMT and multimodal pre-trained models for vision-language tasks, primarily in English, which have shown exceptional generalisation ability.
• HaVQA: A Dataset for Visual Question Answering and Multimodal Research in Hausa Language
  28 May 2023 1 repository listed
  This paper presents HaVQA, the first multimodal dataset for visual question-answering (VQA) tasks in the Hausa language.
• BigVideo: A Large-scale Video Subtitle Translation Dataset for Multimodal Machine Translation
  23 May 2023 1 repository listed
  We also introduce two deliberately designed test sets to verify the necessity of visual information: Ambiguous with the presence of ambiguous words, and Unambiguous in which the text context is self-contained for…
• Scene Graph as Pivoting: Inference-time Image-free Unsupervised Multimodal Machine Translation with Visual Scene Hallucination
  20 May 2023 1 repository listed
  In this work, we investigate a more realistic unsupervised multimodal machine translation (UMMT) setup, inference-time image-free UMMT, where the model is trained with source-text image pairs, and tested with only…
• Beyond Triplet: Leveraging the Most Data for Multimodal Machine Translation
  20 Dec 2022 1 repository listed
  Therefore, this paper correspondingly establishes new methods and new datasets for MMT.
• Distill the Image to Nowhere: Inversion Knowledge Distillation for Multimodal Machine Translation
  10 Oct 2022 1 repository listed Syntology ran 4 of 5 samples · 1 unverified · 5 pointer-only (licence)
  Thus, in this work, we introduce IKD-MMT, a novel MMT framework to support the image-free inference phase via an inversion knowledge distillation scheme.
• VALHALLA: Visual Hallucination for Machine Translation
  31 May 2022 1 repository listed Syntology ran 6 of 10 samples · 4 unverified
  In particular, given a source sentence an autoregressive hallucination transformer is used to predict a discrete visual representation from the input text, and the combined text and hallucinated representations are…
• Neural Machine Translation with Phrase-Level Universal Visual Representations
  19 Mar 2022 1 repository listed Syntology ran 0 of 1 samples · 1 unverified · 1 pointer-only (licence)
  Multimodal machine translation (MMT) aims to improve neural machine translation (NMT) with additional visual information, but most existing MMT methods require paired input of source sentence and image, which makes them…
• MSCTD: A Multimodal Sentiment Chat Translation Dataset
  28 Feb 2022 1 repository listed
  In this work, we introduce a new task named Multimodal Chat Translation (MCT), aiming to generate more accurate translations with the help of the associated dialogue history and visual context.
• VISA: An Ambiguous Subtitles Dataset for Visual Scene-Aware Machine Translation
  20 Jan 2022 1 repository listed
  Existing multimodal machine translation (MMT) datasets consist of images and video captions or general subtitles, which rarely contain linguistic ambiguity, making visual information not so effective to generate…
• Vision Matters When It Should: Sanity Checking Multimodal Machine Translation Models
  8 Sep 2021 1 repository listed
  Multimodal machine translation (MMT) systems have been shown to outperform their text-only neural machine translation (NMT) counterparts when visual context is available.
• BERTGEN: Multi-task Generation through BERT
  7 Jun 2021 1 repository listed
  We present BERTGEN, a novel generative, decoder-only model which extends BERT by fusing multimodal and multilingual pretrained models VL-BERT and M-BERT, respectively.
• ViTA: Visual-Linguistic Translation by Aligning Object Tags
  1 Jun 2021 1 repository listed
  Multimodal Machine Translation (MMT) enriches the source text with visual information for translation.
• Cultural and Geographical Influences on Image Translatability of Words across Languages
  1 Jun 2021 1 repository listed
  We find that images of words are not always invariant across languages, and that language pairs with shared culture, meaning having either a common language family, ethnicity or religion, have improved image…
• Cross-lingual Visual Pre-training for Multimodal Machine Translation
  25 Jan 2021 1 repository listed
  Pre-trained language models have been shown to improve performance in many natural language tasks substantially.
• Dynamic Context-guided Capsule Network for Multimodal Machine Translation
  4 Sep 2020 1 repository listed
  Particularly, we represent the input image with global and regional visual features, we introduce two parallel DCCNs to model multimodal context vectors with visual features at different granularities.
• Multimodal Transformer for Multimodal Machine Translation
  1 Jul 2020 1 repository listed
  Multimodal Machine Translation (MMT) aims to introduce information from other modality, generally static images, to improve the translation quality.
• Self-Knowledge Distillation with Progressive Refinement of Targets
  22 Jun 2020 1 repository listed Syntology ran 4 of 5 samples · 1 unverified
  Hence, it can be interpreted within a framework of knowledge distillation as a student becomes a teacher itself.
• M3P: Learning Universal Representations via Multitask Multilingual Multimodal Pre-training
  4 Jun 2020 1 repository listed
  We present M3P, a Multitask Multilingual Multimodal Pre-trained model that combines multilingual pre-training and multimodal pre-training into a unified framework via multitask pre-training.
• Distilling Translations with Visual Awareness
  18 Jun 2019 1 repository listed
  Previous work on multimodal machine translation has shown that visual information is only needed in very specific cases, for example in the presence of ambiguous words where the textual context is not sufficient.

Syntology lines on 8 of the papers shown; no Syntology record for the others (a paper without an arXiv id cannot be joined to the graph, and absence from the graph layer is not a recorded non-run). “Ran” means the sample executed on a synthesized fixture, not that the paper's result was reproduced. Read from the graph 2026-09-24.