Papers › Enriching Word Vectors with Subword Information

Enriching Word Vectors with Subword Information

15 Jul 2016TACL 2017 1arXiv:1607.04606archive 2025-07-28

Piotr Bojanowski, Edouard Grave, Armand Joulin, Tomas Mikolov

Continuous word representations, trained on large unlabeled corpora are useful for many natural language processing tasks. Popular models that learn such representations ignore the morphology of words, by assigning a distinct vector to each word. This is a limitation, especially for languages with large vocabularies and many rare words. In this paper, we propose a new approach based on the skipgram model, where each word is represented as a bag of character n-grams. A vector representation is associated to each character n-gram; words being represented as the sum of these representations. Our method is fast, allowing to train models on large corpora quickly and allows us to compute word representations for words that did not appear in the training data. We evaluate our word representations on nine different languages, both on word similarity and analogy tasks. By comparing to recently proposed morphological word representations, we show that our vectors achieve state-of-the-art performance on these tasks.

PaperPDFConference PDFCodeCode Syntology ran

In Syntology Open this paper in Syntology's Atlas, the map of the papers in Syntology's graph and their citations.

For agents, Syntology's MCP tool lists every function and class Syntology harvested from this paper and whether it ran (how to connect): get_harvested_code_for_paper(arxiv_id="1607.04606")

Code

Syntology Ran 4 of 31 code samples harvested from 7 repositories linked to this paper; 27 have no recorded run. Of those that ran: 4 ran · our draft was wrong.

By repository: community (archive-listed): 31 samples from 7 repositories, 4 ran. The run record, sample by sample. “Ran” means executed on a synthesized input, not that the code is correct or reproduces the paper.

54 repositories listed; official and paper-mentioned ones first.

facebookresearch/fastText officialmentioned in papermentioned on GitHubMIT report
Babylonpartners/fastText_multilingual mentioned on GitHubBSD-3-Clause report
DW-yejing/fasttext4j-jdk6 mentioned on GitHubBSD-3-Clause report
FengJiaChunFromSYSU/fastText mentioned on GitHubMIT report
GitHubSprint/fasttext4j mentioned on GitHubBSD-3-Clause report
Kinetikm/fastTextRelearning mentioned on GitHub report
Kinetikm/fasttextRelearnExperiment mentioned on GitHubNOASSERTION report
M155K4R4/fastText mentioned on GitHubNOASSERTION report
MhYao2014/HyperbolicEmbedding mentioned on GitHubMIT report
MirkoLenz/ReCAP-Argument-Graph-Retrieval mentioned on GitHubtfApache-2.0 report
Nim-NLP/fastText mentioned on GitHubNOASSERTION report
Omerktn/fastText-iterative mentioned on GitHub report
ReCAP-UTR/Argument-Graph-Retrieval mentioned on GitHubtfApache-2.0 report
SarangShaikh201/fastText mentioned on GitHub report
amymariaparker2401/new mentioned on GitHubNOASSERTION report
babylonhealth/fastText_multilingual mentioned on GitHubBSD-3-Clause report
bamtercelboo/cw2vec mentioned on GitHubApache-2.0 report
bung87/fastText mentioned on GitHubNOASSERTION report
currentsapi/fastlangid mentioned on GitHubApache-2.0 report
dbaumgarten/FToDTF mentioned on GitHubtfApache-2.0 report
divisionai/fastText mentioned on GitHubMIT report
ericxsun/fastText mentioned on GitHubNOASSERTION report
explosion/floret mentioned on GitHubMIT report
hufscapstone/Fast_text mentioned on GitHubMIT report
indix/whatthelang mentioned on GitHubApache-2.0 report
jen1995/fastText mentioned on GitHubMIT report
kpu/fastertext mentioned on GitHubMIT report
labdac/charlacompling mentioned on GitHub report
linkfluence/fastText4j mentioned on GitHubBSD-3-Clause report
lmd1993/fastTextBoost mentioned on GitHubNOASSERTION report
luckyPT/jvm-ml mentioned on GitHubtf report
luhuiguo/jfasttext mentioned on GitHubNOASSERTION report
mrzzy/np-dl-assign-2 mentioned on GitHubtf report
mwydmuch/extremeText mentioned on GitHubtfNOASSERTION report
plasticityai/magnitude mentioned on GitHubpytorchMIT report
pommedeterresautee/fastrtext mentioned on GitHubNOASSERTION report
ppke-nlpg/fastText_factored-cbow mentioned on GitHubMIT report
rmenegaux/fastDNA mentioned on GitHubNOASSERTION report
romik9999/fasttext-1925f09ed3 mentioned on GitHubNOASSERTION report
trietnm2/sent2vec4j mentioned on GitHub report
tshev/faster-FastText mentioned on GitHubMIT report
ulf1/augtxt mentioned on GitHubApache-2.0 report
vinhkhuc/JFastText mentioned on GitHubNOASSERTION report
wyfish/fastText mentioned on GitHubnot reachable when probed 2026-09-18 — repositories for recent papers often appear after camera-ready report
xuzhezhaozhao/fastText_reading mentioned on GitHubNOASSERTION report
zhang2010hao/cw2vec-pytorch mentioned on GitHubpytorch report
cui0523/Code6 mindspore report
dmlc/gluon-nlp mxnetApache-2.0 report

Repository list and official/mentioned flags are the archive's, frozen 2025-07-28. Reachability, where shown, is from one Syntology probe window (2026-09-16 to 2026-09-18); repositories not probed show nothing. GitHub stars are not tracked.

Code Syntology ran Syntology

31 samples harvested; 4 ran; 0 honoured the contract we drafted; 27 have no recorded run. Read from Syntology's graph 2026-09-24; that is when this build read the record, not when the samples ran.

4ran · our draft was wrong
27unverified

Licence: 4 of the 31 samples are pointer only, meaning Syntology does not serve that copy's text. This page shows no code text for any sample; each one links to its file in the repository.

Harvested from 7 repositories linked to this paper, official or community; each sample names its own and says which. “Ran” means the sample executed on a synthesized input. It does not mean the output is correct, and nothing here reproduces the paper's results. “Honoured” and “violated” refer to a contract Syntology drafted from the code itself; “our draft was wrong” and “fixture could not drive it” are failures of Syntology's instrument, not of the code.

Each sample ends with its code_sha256, Syntology's identity for that exact code. An agent fetches the stored sample with Syntology's MCP tool get_code(code_sha256="…") (how to connect); click an identity to copy that call.

Repository labels, per sample. official repository: The archive marks this repository official for the paper. named in the paper: The archive records that the paper mentions this repository; it is not marked official. community (archive-listed): In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper. found in paper text by Syntology: Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted. community: Not in the archive's code links for this paper; a community repository Syntology harvested. Samples from a repository marked official are listed first. Licence labels name the repository's licence as recorded at harvest. “Pointer only” means Syntology does not serve that copy's text, for one of four reasons: no licence file was found; the licence was not identified; the licence is recorded as permissive but that copy's record is not marked cleared; or the licence is outside the permissive list Syntology serves text under (MIT, Apache-2.0, BSD and similar). Some licences outside that list permit redistribution, such as WTFPL, and GPL-3.0 under its conditions; they are simply not on the list. Hover a licence label for the reason. File links open the file on GitHub at the default branch, which may have changed since the harvest.

create_lookup_tables SarangShaikh201/fastText/utils.py community (archive-listed) ran · our draft was wrong fingerprinted licence not identified · pointer only · a16b79c7bc1860e5 · report
load_model zhang2010hao/cw2vec-pytorch/cw2vec.py community (archive-listed) ran · our draft was wrong no licence file found · pointer only · 847e1524ed1bff1a · report
preprocess SarangShaikh201/fastText/utils.py community (archive-listed) ran · our draft was wrong fingerprinted licence not identified · pointer only · 55d2494d83e5c523 · report
read_args Omerktn/fastText-iterative/python/fasttext_module/fasttext/FastText.py community (archive-listed) ran · our draft was wrong MIT (permissive) · f697966cbe929024 · report
char_ngrams plasticityai/magnitude/pymagnitude/converter_shared.py community (archive-listed) unverified MIT (permissive) · 478a1a9fc517ce62 · report
check_glibc_version plasticityai/magnitude/glibc.py community (archive-listed) unverified MIT (permissive) · 39f3e4d5f2279ff9 · report
compute_similarity Kinetikm/fastTextRelearning/python/doc/examples/train_unsupervised.py community (archive-listed) unverified MIT (permissive) · 411c04ff52be0d9f · report
compute_word_similarities dbaumgarten/FToDTF/ftodtf/model.py community (archive-listed) unverified Apache-2.0 (permissive) · 153d1c4c32072113 · report
draw_index ulf1/augtxt/augtxt/typo.py community (archive-listed) unverified Apache-2.0 (permissive) · 4464b010ea7987c2 · report
drop_word ulf1/augtxt/augtxt/order.py community (archive-listed) unverified Apache-2.0 (permissive) · 839d40e2e8ad110b · report
entropy plasticityai/magnitude/pymagnitude/converter.py community (archive-listed) unverified MIT (permissive) · 6e32ac79f67582c1 · report
fast_md5_file plasticityai/magnitude/pymagnitude/converter_shared.py community (archive-listed) unverified MIT (permissive) · dee6b2dc58e092d3 · report
find_index ulf1/augtxt/augtxt/keyboard_layouts.py community (archive-listed) unverified Apache-2.0 (permissive) · 8742e7667cc1dafa · report
generate_ngram_per_word dbaumgarten/FToDTF/ftodtf/input.py community (archive-listed) unverified Apache-2.0 (permissive) · b5b7c0a5146de532 · report
get_batches SarangShaikh201/fastText/utils.py community (archive-listed) unverified licence not identified · pointer only · cde234836e3f3978 · report
get_config_var plasticityai/magnitude/pep425tags.py community (archive-listed) unverified MIT (permissive) · f546d4abc08099d7 · report
get_darwin_arches plasticityai/magnitude/pep425tags.py community (archive-listed) unverified MIT (permissive) · e3ecbfdb8037b64d · report
get_flag plasticityai/magnitude/pep425tags.py community (archive-listed) unverified MIT (permissive) · 5066f582560cf77c · report
have_compatible_glibc plasticityai/magnitude/glibc.py community (archive-listed) unverified MIT (permissive) · e0fd811edefc09ba · report
md5_file plasticityai/magnitude/pymagnitude/converter_shared.py community (archive-listed) unverified MIT (permissive) · a89e48d6397ae9ba · report
merge_words ulf1/augtxt/augtxt/punct.py community (archive-listed) unverified Apache-2.0 (permissive) · 280281e946bcacff · report
ngrams_to_vectors dbaumgarten/FToDTF/ftodtf/model.py community (archive-listed) unverified Apache-2.0 (permissive) · cf3407d8e4174215 · report
pad_to_length dbaumgarten/FToDTF/ftodtf/input.py community (archive-listed) unverified Apache-2.0 (permissive) · fa43a85dd995968e · report
parse_batch_func dbaumgarten/FToDTF/ftodtf/model.py community (archive-listed) unverified Apache-2.0 (permissive) · 00bd40e0432ca4e5 · report
pressed_twice ulf1/augtxt/augtxt/typo.py community (archive-listed) unverified Apache-2.0 (permissive) · ff4bb2b5311e1044 · report
random_args ulf1/augtxt/augtxt/augmenters.py community (archive-listed) unverified Apache-2.0 (permissive) · 4f5bb4400e32b7e9 · report
remove_syntaxinfo ulf1/augtxt/augtxt/punct.py community (archive-listed) unverified Apache-2.0 (permissive) · 37d343258aa5993a · report
swap_consecutive ulf1/augtxt/augtxt/order.py community (archive-listed) unverified Apache-2.0 (permissive) · 2eb54f96aa23c052 · report
swap_consecutive ulf1/augtxt/augtxt/typo.py community (archive-listed) unverified Apache-2.0 (permissive) · 48776fc059b48990 · report
synonym_replacement ulf1/augtxt/augtxt/wordsubs.py community (archive-listed) unverified Apache-2.0 (permissive) · e36a7da38a737dd2 · report
write_twice ulf1/augtxt/augtxt/order.py community (archive-listed) unverified Apache-2.0 (permissive) · 5f310db16b7971c6 · report

Tasks

Word EmbeddingsWord Similarity

Results from the paper archive 2025-07-28

TaskDatasetModelMetricValueRank at snapshotLeaderboardReport
Word Similarity WS353 SkipGram Spearman's Rho 61.0 #3 of 3 Archive leaderboard report

Ranks are positions in the archive's leaderboards as they stood at the 2025-07-28 snapshot. Results published since then are not among these rows, so a rank here is not a current standing.

Methods

Introduced by this paper: fastText

fastText

Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections