Papers › A Latent Variable Model Approach to PMI-based Word Embeddings

A Latent Variable Model Approach to PMI-based Word Embeddings

12 Feb 2015TACL 2016 1arXiv:1502.03520archive 2025-07-28

Sanjeev Arora, Yuanzhi Li, YIngyu Liang, Tengyu Ma, Andrej Risteski

Semantic word embeddings represent the meaning of a word via a vector, and are created by diverse methods. Many use nonlinear operations on co-occurrence statistics, and have hand-tuned hyperparameters and reweighting methods. This paper proposes a new generative model, a dynamic version of the log-linear topic model of. The methodological novelty is to use the prior to compute closed form expressions for word statistics. This provides a theoretical justification for nonlinear models like PMI, word2vec, and GloVe, as well as some hyperparameter choices. It also helps explain why low-dimensional semantic embeddings contain linear algebraic structure that allows solution of word analogies, as shown by and many subsequent papers. Experimental support is provided for the generative model assumptions, the most important of which is that latent word vectors are fairly uniformly dispersed in space.

PaperPDFConference PDFCode

In Syntology View this paper on Syntology, its page in Syntology's graph. That page lists the repositories linked to the paper, the abstract and the calls for agents.

Open this paper in Syntology's Atlas, the map of the papers in Syntology's graph and their citations.

For agents, on Syntology's MCP service (how to connect):

Code

PrincetonML/SemanticVector officialmentioned in papermentioned on GitHubSyntology: not harvested report
LivNLP/Relational-Walk-for-Knowledge-Graphs mentioned on GitHubtfSyntology: not harvested report
tsandefer/capstone_2 mentioned on GitHubtfSyntology: not harvested report
tsandefer/dsi_capstone_2 mentioned on GitHubtfSyntology: not harvested report

Repository list and official/mentioned flags are the archive's, frozen 2025-07-28. Reachability, where shown, is from one Syntology probe window (2026-09-16 to 2026-09-18); repositories not probed show nothing. GitHub stars are not tracked.

Code Syntology ran Syntology

Not run by Syntology. Nothing on this page verifies that the listed code works.

Tasks

Word Embeddings

Results from the paper archive 2025-07-28

No leaderboard rows for this paper in the archive.

Methods

GloVe

Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections