{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/understand-functionality-and-dimensionality","title":"Understand Functionality and Dimensionality of Vector Embeddings: the Distributional Hypothesis, the Pairwise Inner Product Loss and Its Bias-Variance Trade-off","arxiv_id":"1803.00502","date":"2018-03-01","proceeding":null,"authors":["Zi Yin"],"abstract":"Vector embedding is a foundational building block of many deep learning\nmodels, especially in natural language processing. In this paper, we present a\ntheoretical framework for understanding the effect of dimensionality on vector\nembeddings. We observe that the distributional hypothesis, a governing\nprinciple of statistical semantics, requires a natural unitary-invariance for\nvector embeddings. Motivated by the unitary-invariance observation, we propose\nthe Pairwise Inner Product (PIP) loss, a unitary-invariant metric on the\nsimilarity between two embeddings. We demonstrate that the PIP loss captures\nthe difference in functionality between embeddings, and that the PIP loss is\ntightly connect with two basic properties of vector embeddings, namely\nsimilarity and compositionality. By formulating the embedding training process\nas matrix factorization with noise, we reveal a fundamental bias-variance\ntrade-off between the signal spectrum and noise power in the dimensionality\nselection process. This bias-variance trade-off sheds light on many empirical\nobservations which have not been thoroughly explained, for example the\nexistence of an optimal dimensionality. Moreover, we discover two new results\nabout vector embeddings, namely their robustness against over-parametrization\nand their forward stability. The bias-variance trade-off of the PIP loss\nexplicitly answers the fundamental open problem of dimensionality selection for\nvector embeddings.","url_abs":"http://arxiv.org/abs/1803.00502v4","url_pdf":"http://arxiv.org/pdf/1803.00502v4.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"understand-functionality-and-dimensionality","repo_url":"https://github.com/aaaasssddf/PIP-experiments","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"tf","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":null,"mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}