{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/beyond-normal-on-the-evaluation-of-mutual-1","title":"Beyond Normal: On the Evaluation of Mutual Information Estimators","arxiv_id":"2306.11078","date":"2023-06-19","proceeding":"NeurIPS 2023 11","authors":["Paweł Czyż","Frederic Grabowski","Julia E. Vogt","Niko Beerenwinkel","Alexander Marx"],"abstract":"Mutual information is a general statistical dependency measure which has found applications in representation learning, causality, domain generalization and computational biology. However, mutual information estimators are typically evaluated on simple families of probability distributions, namely multivariate normal distribution and selected distributions with one-dimensional random variables. In this paper, we show how to construct a diverse family of distributions with known ground-truth mutual information and propose a language-independent benchmarking platform for mutual information estimators. We discuss the general applicability and limitations of classical and neural estimators in settings involving high dimensions, sparse interactions, long-tailed distributions, and high mutual information. Finally, we provide guidelines for practitioners on how to select appropriate estimator adapted to the difficulty of problem considered and issues one needs to consider when applying an estimator to a new data set.","url_abs":"https://arxiv.org/abs/2306.11078v2","url_pdf":"https://arxiv.org/pdf/2306.11078v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"beyond-normal-on-the-evaluation-of-mutual-1","repo_url":"https://github.com/cbg-ethz/bmi","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"jax","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"beyond-normal-on-the-evaluation-of-mutual-1","repo_url":"https://github.com/MustaphaBounoua/minde","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"jax","reach":{"status":"ok"}}],"tasks":[{"task_slug":"benchmarking","task_name":"Benchmarking"},{"task_slug":"domain-generalization","task_name":"Domain Generalization"},{"task_slug":"representation-learning","task_name":"Representation Learning"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":null,"atlas_url":"https://app.syntology.ai/?focus=2306.11078","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2306.11078"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/cbg-ethz/bmi","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/MustaphaBounoua/minde","reach":{"status":"ok"}}],"summary":{"unverified":3},"by_repo_kind":{"official":{"samples":3,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"cd37f2102b0ffd3c","entry":"add_noise","repo":"cbg-ethz/bmi","repo_kind":"official","path":"src/bmi/utils.py","file_url":"https://github.com/cbg-ethz/bmi/blob/HEAD/src/bmi/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"cd37f2102b0ffd3c"}},{"code_sha256_prefix":"60e9ca1c2fe30a61","entry":"corr","repo":"cbg-ethz/bmi","repo_kind":"official","path":"src/bmi/estimators/correlation.py","file_url":"https://github.com/cbg-ethz/bmi/blob/HEAD/src/bmi/estimators/correlation.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"60e9ca1c2fe30a61"}},{"code_sha256_prefix":"938330dee8b2ab92","entry":"mi_gauss","repo":"cbg-ethz/bmi","repo_kind":"official","path":"src/bmi/estimators/correlation.py","file_url":"https://github.com/cbg-ethz/bmi/blob/HEAD/src/bmi/estimators/correlation.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"938330dee8b2ab92"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}