{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/demystifying-fixed-k-nearest-neighbor","title":"Demystifying Fixed k-Nearest Neighbor Information Estimators","arxiv_id":"1604.03006","date":"2016-04-11","proceeding":null,"authors":["Weihao Gao","Sewoong Oh","Pramod Viswanath"],"abstract":"Estimating mutual information from i.i.d. samples drawn from an unknown joint\ndensity function is a basic statistical problem of broad interest with\nmultitudinous applications. The most popular estimator is one proposed by\nKraskov and St\\\"ogbauer and Grassberger (KSG) in 2004, and is nonparametric and\nbased on the distances of each sample to its $k^{\\rm th}$ nearest neighboring\nsample, where $k$ is a fixed small integer. Despite its widespread use (part of\nscientific software packages), theoretical properties of this estimator have\nbeen largely unexplored. In this paper we demonstrate that the estimator is\nconsistent and also identify an upper bound on the rate of convergence of the\nbias as a function of number of samples. We argue that the superior performance\nbenefits of the KSG estimator stems from a curious \"correlation boosting\"\neffect and build on this intuition to modify the KSG estimator in novel ways to\nconstruct a superior estimator. As a byproduct of our investigations, we obtain\nnearly tight rates of convergence of the $\\ell_2$ error of the well known fixed\n$k$ nearest neighbor estimator of differential entropy by Kozachenko and\nLeonenko.","url_abs":"http://arxiv.org/abs/1604.03006v2","url_pdf":"http://arxiv.org/pdf/1604.03006v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"demystifying-fixed-k-nearest-neighbor","repo_url":"https://github.com/wgao9/knnie","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok"}}],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/1604.03006","atlas_url":"https://app.syntology.ai/?focus=1604.03006","mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}