{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/estimating-mutual-information-for-discrete","title":"Estimating Mutual Information for Discrete-Continuous Mixtures","arxiv_id":"1709.06212","date":"2017-09-19","proceeding":"NeurIPS 2017 12","authors":["Weihao Gao","Sreeram Kannan","Sewoong Oh","Pramod Viswanath"],"abstract":"Estimating mutual information from observed samples is a basic primitive,\nuseful in several machine learning tasks including correlation mining,\ninformation bottleneck clustering, learning a Chow-Liu tree, and conditional\nindependence testing in (causal) graphical models. While mutual information is\na well-defined quantity in general probability spaces, existing estimators can\nonly handle two special cases of purely discrete or purely continuous pairs of\nrandom variables. The main challenge is that these methods first estimate the\n(differential) entropies of X, Y and the pair (X;Y) and add them up with\nappropriate signs to get an estimate of the mutual information. These\n3H-estimators cannot be applied in general mixture spaces, where entropy is not\nwell-defined. In this paper, we design a novel estimator for mutual information\nof discrete-continuous mixtures. We prove that the proposed estimator is\nconsistent. We provide numerical experiments suggesting superiority of the\nproposed estimator compared to other heuristics of adding small continuous\nnoise to all the samples and applying standard estimators tailored for purely\ncontinuous variables, and quantizing the samples and applying standard\nestimators tailored for purely discrete variables. This significantly widens\nthe applicability of mutual information estimation in real-world applications,\nwhere some variables are discrete, some continuous, and others are a mixture\nbetween continuous and discrete components.","url_abs":"http://arxiv.org/abs/1709.06212v3","url_pdf":"http://arxiv.org/pdf/1709.06212v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"estimating-mutual-information-for-discrete","repo_url":"https://github.com/alexandreguichet/MFS","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":{"status":"unanswered"}}],"tasks":[{"task_slug":"clustering","task_name":"Clustering"},{"task_slug":"mutual-information-estimation","task_name":"Mutual Information Estimation"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=1709.06212","mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}