Papers › On the Discrepancy between Density Estimation and Sequence Generation

On the Discrepancy between Density Estimation and Sequence Generation

17 Feb 2020EMNLP (spnlp) 2020 11arXiv:2002.07233archive 2025-07-28

Jason Lee, Dustin Tran, Orhan Firat, Kyunghyun Cho

Many sequence-to-sequence generation tasks, including machine translation and text-to-speech, can be posed as estimating the density of the output y given the input x: p(y|x). Given this interpretation, it is natural to evaluate sequence-to-sequence models using conditional log-likelihood on a test set. However, the goal of sequence-to-sequence generation (or structured prediction) is to find the best output y^ given an input x, and each task has its own downstream metric R that scores a model output by comparing against a set of references y*: R(y^, y* | x). While we hope that a model that excels in density estimation also performs well on the downstream metric, the exact correlation has not been studied for sequence generation tasks. In this paper, by comparing several density estimators on five machine translation tasks, we find that the correlation between rankings of models based on log-likelihood and BLEU varies significantly depending on the range of the model families being compared. First, log-likelihood is highly correlated with BLEU when we consider models within the same family (e.g. autoregressive models, or latent variable models with the same parameterization of the prior). However, we observe no correlation between rankings of models across different families: (1) among non-autoregressive latent variable models, a flexible prior distribution is better at density estimation but gives worse generation quality than a simple prior, and (2) autoregressive models offer the best translation performance overall, while latent variable models with a normalizing flow prior give the highest held-out log-likelihood across all datasets. Therefore, we recommend using a simple prior for the latent variable non-autoregressive model when fast generation speed is desired.

PaperPDFConference PDFCode

In Syntology View this paper on Syntology, its page in Syntology's graph. That page lists the repositories linked to the paper, the abstract and the calls for agents.

Open this paper in Syntology's Atlas, the map of the papers in Syntology's graph and their citations.

For agents, on Syntology's MCP service (how to connect):

Code

tensorflow/tensor2tensor officialmentioned in papertfSyntology: no sample linked to this paper (harvested for another paper). Syntology's graph links these samples to Tensor2Tensor for Neural Machine Translation (arXiv:1803.07416; that paper's own run record: 30 ran (of which 0 constructed an object rather than computing a result; 29 with no instrument failure: 0 honoured, 0 violated, 29 with no contract checked; 1 where Syntology's instrument failed) · 18 unverified), Model-Based Reinforcement Learning for Atari (arXiv:1903.00374; that paper's own run record: 16 ran (of which 8 constructed an object rather than computing a result; 10 with no instrument failure: 0 honoured, 0 violated, 10 with no contract checked; 6 where Syntology's instrument failed) · 5 unverified) and Simplify-then-Translate: Automatic Preprocessing for Black-Box Machine Translation (arXiv:2005.11197; that paper's own run record: 2 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 2 where Syntology's instrument failed) · 2 unverified); nothing checks that they implement this paper's method. report

Repository list and official/mentioned flags are the archive's, frozen 2025-07-28. Reachability, where shown, is from one Syntology probe window (2026-09-16 to 2026-09-18); repositories not probed show nothing. GitHub stars are not tracked.

A paper named beside a repository with no sample linked to this paper is shown with that paper's own run record, not this paper's: “ran” means executed on a synthesized input, not that the code is correct, and “where Syntology's instrument failed” counts failures of Syntology's instrument, not of the code. Papers are listed in arXiv-id order, at most three per repository.

Code Syntology ran Syntology

Not run by Syntology. Nothing on this page verifies that the listed code works.

Tasks

Density EstimationMachine TranslationStructured PredictionText to SpeechTranslationtext-to-speech

Results from the paper archive 2025-07-28

No leaderboard rows for this paper in the archive.

Methods

SPEEDTest

Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections