{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/estimating-mutual-information","title":"Estimating Mutual Information","arxiv_id":"cond-mat/0305641","date":"2003-05-28","proceeding":null,"authors":["Alexander Kraskov","Harald Stoegbauer","Peter Grassberger"],"abstract":"We present two classes of improved estimators for mutual information $M(X,Y)$, from samples of random points distributed according to some joint probability density $\\mu(x,y)$. In contrast to conventional estimators based on binnings, they are based on entropy estimates from $k$-nearest neighbour distances. This means that they are data efficient (with $k=1$ we resolve structures down to the smallest possible scales), adaptive (the resolution is higher where data are more numerous), and have minimal bias. Indeed, the bias of the underlying entropy estimates is mainly due to non-uniformity of the density at the smallest resolved scale, giving typically systematic errors which scale as functions of $k/N$ for $N$ points. Numerically, we find that both families become {\\it exact} for independent distributions, i.e. the estimator $\\hat M(X,Y)$ vanishes (up to statistical fluctuations) if $\\mu(x,y) = \\mu(x) \\mu(y)$. This holds for all tested marginal distributions and for all dimensions of $x$ and $y$. In addition, we give estimators for redundancies between more than 2 random variables. We compare our algorithms in detail with existing algorithms. Finally, we demonstrate the usefulness of our estimators for assessing the actual independence of components obtained from independent component analysis (ICA), for improving ICA, and for estimating the reliability of blind source separation.","url_abs":"https://arxiv.org/abs/cond-mat/0305641v1","url_pdf":"https://arxiv.org/pdf/cond-mat/0305641v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"links_only","authors_date_abstract":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license), from the Kaggle arXiv metadata snapshot of 2026-09-12"},"code_links":[{"paper_slug":"estimating-mutual-information","repo_url":"https://github.com/albanyhep/JSDML","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"tf","reach":{"status":"ok"}},{"paper_slug":"estimating-mutual-information","repo_url":"https://github.com/alexandreguichet/MFS","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":{"status":"unanswered"}},{"paper_slug":"estimating-mutual-information","repo_url":"https://github.com/cbg-ethz/bmi","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"jax","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"estimating-mutual-information","repo_url":"https://github.com/mutualinfo/mutual_info","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok","spdx":"Apache-2.0"}},{"paper_slug":"estimating-mutual-information","repo_url":"https://github.com/nordavinden/Mutual-Information-ICA","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok","spdx":"GPL-3.0"}},{"paper_slug":"estimating-mutual-information","repo_url":"https://github.com/paulbrodersen/entropy_estimators","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok","spdx":"GPL-3.0"}}],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=cond-mat%2F0305641","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"cond-mat/0305641"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/cbg-ethz/bmi","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/paulbrodersen/entropy_estimators","reach":{"status":"ok","spdx":"GPL-3.0"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/mutualinfo/mutual_info","reach":{"status":"ok","spdx":"Apache-2.0"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/albanyhep/JSDML","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/alexandreguichet/MFS","reach":{"status":"unanswered"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/nordavinden/Mutual-Information-ICA","reach":{"status":"ok","spdx":"GPL-3.0"}}],"summary":{"unverified":3},"by_repo_kind":{"listed":{"samples":3,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"e0e1881c92995082","entry":"handle_transform","repo":"mutualinfo/mutual_info","repo_kind":"listed","path":"mutual_info/mutual_info.py","file_url":"https://github.com/mutualinfo/mutual_info/blob/HEAD/mutual_info/mutual_info.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"e0e1881c92995082"}},{"code_sha256_prefix":"b139b834126912a3","entry":"rank_transform","repo":"mutualinfo/mutual_info","repo_kind":"listed","path":"mutual_info/mutual_info.py","file_url":"https://github.com/mutualinfo/mutual_info/blob/HEAD/mutual_info/mutual_info.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"b139b834126912a3"}},{"code_sha256_prefix":"35f8e155803950cb","entry":"standardize_transform","repo":"mutualinfo/mutual_info","repo_kind":"listed","path":"mutual_info/mutual_info.py","file_url":"https://github.com/mutualinfo/mutual_info/blob/HEAD/mutual_info/mutual_info.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"35f8e155803950cb"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}