Methods › Natural Language Processing › Subword Segmentation › Unigram Segmentation
Unigram Segmentation
Introduced by Taku Kudo in Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidates
archive 2025-07-28 Description, source and code snippet are the archive's method entry.
Unigram Segmentation is a subword segmentation algorithm based on a unigram language model. It provides multiple segmentations with probabilities. The language model allows for emulating the noise generated during the segmentation of actual data.
The unigram language model makes an assumption that each subword occurs independently, and consequently, the probability of a subword sequence 𝐱 = (x₁,…,x_M) is formulated as the product of the subword occurrence probabilities p(xᵢ):
P(𝐱) = ∏ᵢ₌₁ᴹ p(xᵢ), ∀i xᵢ ∈𝒱, ∑_(x ∈𝒱) p(x) = 1,
where 𝒱 is a pre-determined vocabulary. The most probable segmentation 𝐱^* for the input sentence X is then given by:
𝐱^* = argmax_(𝐱 ∈𝒮(X)) P(𝐱),
where 𝒮(X) is a set of segmentation candidates built from the input sentence X. 𝐱^* is obtained with the Viterbi algorithm.
Papers archive 2025-07-28
1 shown of 1, newest first. Repository counts are the archive's code-links table. A Syntology line states what Syntology ran from that paper's harvested code; it is per sample and not a correctness claim.
-
Benchmarking Azerbaijani Neural Machine Translation 29 Jul 2022 · 0 repositories · arXiv:2207.14473
Tasks archive 2025-07-28
6 tasks the archive attaches to papers tagged with this method, by distinct papers. A task without a page in the catalog is plain text.
| Task | Papers |
|---|---|
| Benchmarking | 1 |
| Domain Generalization | 1 |
| Machine Translation | 1 |
| NMT | 1 |
| Segmentation | 1 |
| Translation | 1 |
Usage over time archive 2025-07-28
Components: the archive holds no method-to-method composition, so PwC's Components table cannot be rebuilt; the Papers list carries no Results column for the same reason (the archive does not join its leaderboard rows to method tags).
Categories archive 2025-07-28
Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections