{"url":"/method/levenshtein-transformer","slug":"levenshtein-transformer","name":"Levenshtein Transformer","full_name":"Levenshtein Transformer","full_name_withheld":false,"description_markdown":"The **Levenshtein Transformer** (LevT) is a type of [transformer](https://paperswithcode.com/method/transformer) that aims to address the lack of flexibility of previous decoding models. Notably, in previous frameworks, the length of generated sequences is either fixed or monotonically increased as the decoding proceeds. The authors argue this is incompatible with human-level intelligence where humans can revise, replace, revoke or delete any part of their generated text. Hence, LevT is proposed to bridge this gap by breaking the in-so-far standardized decoding mechanism and replacing it with two basic operations — insertion and deletion.\r\n\r\nLevT is trained using imitation learning. The resulted model contains two policies and they are executed in an alternate manner. The authors argue that with this model decoding becomes more flexible. For example, when the decoder is given an empty token, it falls back to a normal sequence generation model. On the other hand, the decoder acts as a refinement model when the initial state is a low-quality generated sequence.\r\n\r\nOne crucial component in LevT framework is the learning algorithm. The authors leverage the characteristics of insertion and deletion — they are complementary but also adversarial. The algorithm they propose is called “dual policy learning”. The idea is that when training one policy (insertion or deletion), we use the output from its adversary at the previous iteration as input. An expert policy, on the other hand, is drawn to provide a correction signal.","description_state":"present","introduced_year":null,"introduced_by":{"title":"Levenshtein Transformer","paper":"/paper/levenshtein-transformer","first_author":"Jiatao Gu","n_authors":3,"url_abs":null,"archive_paper_url":"https://paperswithcode.com/paper/levenshtein-transformer"},"source":{"url":"https://arxiv.org/abs/1905.11006v2","title":"Levenshtein Transformer","url_on_a_paper_host":true},"code_snippet_url":"https://github.com/pytorch/fairseq/blob/28876638114948711fd4bd4e350fdd6809013f1e/fairseq/models/nat/levenshtein_transformer.py#L34","code_snippet_url_on_a_code_host":true,"categories":[{"area":"Natural Language Processing","area_id":"natural-language-processing","collection":"Autoregressive Transformers","url":"/methods/category/autoregressive-transformers","pwc_aliases":[]},{"area":"Natural Language Processing","area_id":"natural-language-processing","collection":"Transformers","url":"/methods/category/transformers","pwc_aliases":[]}],"n_papers_tagged":12,"archive_num_papers":12,"papers_newest_first":[{"paper":null,"title":"Reinforcement Learning for Edit-Based Non-Autoregressive Neural Machine Translation","date":"2024-05-02","arxiv_id":"2405.01280","n_code_links":0,"syntology":null},{"paper":"/paper/towards-example-based-nmt-with-multi","title":"Towards Example-Based NMT with Multi-Levenshtein Transformers","date":"2023-10-13","arxiv_id":"2310.08967","n_code_links":1,"syntology":{"ran":1,"of":1,"unverified":0,"pointer_only":0}},{"paper":"/paper/non-autoregressive-machine-translation-with-2","title":"Integrating Translation Memories into Non-Autoregressive Machine Translation","date":"2022-10-12","arxiv_id":"2210.06020","n_code_links":1,"syntology":null},{"paper":"/paper/levenshtein-ocr","title":"Levenshtein OCR","date":"2022-09-08","arxiv_id":"2209.03594","n_code_links":2,"syntology":null},{"paper":null,"title":"The JHU-Microsoft Submission for WMT21 Quality Estimation Shared Task","date":"2021-09-17","arxiv_id":"2109.08724","n_code_links":0,"syntology":null},{"paper":"/paper/levenshtein-training-for-word-level-quality","title":"Levenshtein Training for Word-level Quality Estimation","date":"2021-09-12","arxiv_id":"2109.05611","n_code_links":1,"syntology":null},{"paper":null,"title":"Using Perturbed Length-aware Positional Encoding for Non-autoregressive Neural Machine Translation","date":"2021-07-29","arxiv_id":"2107.13689","n_code_links":0,"syntology":null},{"paper":null,"title":"How Does Distilled Data Complexity Impact the Quality and Confidence of Non-Autoregressive Machine Translation?","date":"2021-05-27","arxiv_id":"2105.12900","n_code_links":0,"syntology":null},{"paper":"/paper/editor-an-edit-based-transformer-with","title":"EDITOR: an Edit-Based Transformer with Repositioning for Neural Machine Translation with Soft Lexical Constraints","date":"2020-11-13","arxiv_id":"2011.06868","n_code_links":1,"syntology":null},{"paper":null,"title":"Expand and Filter: CUNI and LMU Systems for the WNGT 2020 Duolingo Shared Task","date":"2020-07-01","arxiv_id":null,"n_code_links":0,"syntology":null},{"paper":"/paper/lexically-constrained-neural-machine","title":"Lexically Constrained Neural Machine Translation with Levenshtein Transformer","date":"2020-04-27","arxiv_id":"2004.12681","n_code_links":1,"syntology":{"ran":1,"of":4,"unverified":3,"pointer_only":4}},{"paper":"/paper/levenshtein-transformer","title":"Levenshtein Transformer","date":"2019-05-27","arxiv_id":"1905.11006","n_code_links":3,"syntology":null}],"papers_shown":12,"tasks":[{"task":"/task/translation","name":"Translation","papers":11},{"task":"/task/machine-translation","name":"Machine Translation","papers":8},{"task":"/task/imitation-learning","name":"Imitation Learning","papers":3},{"task":"/task/knowledge-distillation","name":"Knowledge Distillation","papers":3},{"task":"/task/diversity","name":"Diversity","papers":2},{"task":"/task/nmt","name":"NMT","papers":2},{"task":"/task/automatic-post-editing","name":"Automatic Post-Editing","papers":1},{"task":"/task/data-augmentation","name":"Data Augmentation","papers":1},{"task":"/task/domain-adaptation","name":"Domain Adaptation","papers":1},{"task":"/task/optical-character-recognition","name":"Optical Character Recognition (OCR)","papers":1},{"task":"/task/reinforcement-learning-1","name":"Reinforcement Learning (RL)","papers":1},{"task":"/task/retrieval","name":"Retrieval","papers":1},{"task":"/task/scene-text-recognition","name":"Scene Text Recognition","papers":1},{"task":"/task/task-2","name":"Task 2","papers":1},{"task":"/task/text-summarization","name":"Text Summarization","papers":1},{"task":"/task/transfer-learning","name":"Transfer Learning","papers":1},{"task":"/task/reinforcement-learning-2","name":"reinforcement-learning","papers":1}],"tasks_shown":17,"n_tasks":17,"usage_by_year":[{"year":"2019","papers":1},{"year":"2020","papers":3},{"year":"2021","papers":4},{"year":"2022","papers":2},{"year":"2023","papers":1},{"year":"2024","papers":1}],"row_source":"methods_table","archive":{"source":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","archive_url":"https://paperswithcode.com/method/levenshtein-transformer"},"syntology_read_at":"2026-09-24T18:15:14+00:00"}