{"url":"/method/unigram-segmentation","slug":"unigram-segmentation","name":"Unigram Segmentation","full_name":"Unigram Segmentation","full_name_withheld":false,"description_markdown":"**Unigram Segmentation** is a subword segmentation algorithm based on a unigram language model. It provides multiple segmentations with probabilities. The language model allows for emulating the noise generated during the segmentation of actual data.\r\n\r\nThe unigram language model makes an assumption that each subword occurs independently, and consequently, the probability of a subword sequence $\\mathbf{x} = (x_1,\\ldots,x_M)$ is\r\nformulated as the product of the subword occurrence probabilities\r\n$p(x_i)$:\r\n\r\n$$\r\n  P(\\mathbf{x}) = \\prod_{i=1}^{M} p(x_i), \\\\\\\\\r\n  \\forall i\\,\\, x_i \\in \\mathcal{V},\\,\\,\\,\r\n  \\sum_{x \\in \\mathcal{V}} p(x) = 1, \\nonumber\r\n$$\r\n\r\nwhere $\\mathcal{V}$ is a pre-determined vocabulary.  The most probable\r\nsegmentation $\\mathbf{x}^*$ for the input sentence $X$ is then given by:\r\n\r\n$$\r\n  \\mathbf{x}^{*} = \\text{argmax}_{\\mathbf{x} \\in \\mathcal{S}(X)} P(\\mathbf{x}),\r\n$$\r\n\r\nwhere $\\mathcal{S}(X)$ is a set of segmentation candidates built from\r\nthe input sentence $X$.  $\\mathbf{x}^*$ is obtained with the Viterbi\r\nalgorithm.","description_state":"present","introduced_year":null,"introduced_by":{"title":"Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidates","paper":"/paper/subword-regularization-improving-neural","first_author":"Taku Kudo","n_authors":1,"url_abs":null,"archive_paper_url":"https://paperswithcode.com/paper/subword-regularization-improving-neural"},"source":{"url":"http://arxiv.org/abs/1804.10959v1","title":"Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidates","url_on_a_paper_host":true},"code_snippet_url":null,"code_snippet_url_on_a_code_host":false,"categories":[{"area":"Natural Language Processing","area_id":"natural-language-processing","collection":"Subword Segmentation","url":"/methods/category/subword-segmentation","pwc_aliases":[]}],"n_papers_tagged":1,"archive_num_papers":1,"papers_newest_first":[{"paper":null,"title":"Benchmarking Azerbaijani Neural Machine Translation","date":"2022-07-29","arxiv_id":"2207.14473","n_code_links":0,"syntology":null}],"papers_shown":1,"tasks":[{"task":"/task/benchmarking","name":"Benchmarking","papers":1},{"task":"/task/domain-generalization","name":"Domain Generalization","papers":1},{"task":"/task/machine-translation","name":"Machine Translation","papers":1},{"task":"/task/nmt","name":"NMT","papers":1},{"task":"/task/segmentation","name":"Segmentation","papers":1},{"task":"/task/translation","name":"Translation","papers":1}],"tasks_shown":6,"n_tasks":6,"usage_by_year":[{"year":"2022","papers":1}],"row_source":"methods_table","archive":{"source":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","archive_url":"https://paperswithcode.com/method/unigram-segmentation"},"syntology_read_at":"2026-09-24T18:15:14+00:00"}