{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/club-a-contrastive-log-ratio-upper-bound-of","title":"CLUB: A Contrastive Log-ratio Upper Bound of Mutual Information","arxiv_id":"2006.12013","date":"2020-06-22","proceeding":"ICML 2020 1","authors":["Pengyu Cheng","Weituo Hao","Shuyang Dai","Jiachang Liu","Zhe Gan","Lawrence Carin"],"abstract":"Mutual information (MI) minimization has gained considerable interests in various machine learning tasks. However, estimating and minimizing MI in high-dimensional spaces remains a challenging problem, especially when only samples, rather than distribution forms, are accessible. Previous works mainly focus on MI lower bound approximation, which is not applicable to MI minimization problems. In this paper, we propose a novel Contrastive Log-ratio Upper Bound (CLUB) of mutual information. We provide a theoretical analysis of the properties of CLUB and its variational approximation. Based on this upper bound, we introduce a MI minimization training scheme and further accelerate it with a negative sampling strategy. Simulation studies on Gaussian distributions show the reliable estimation ability of CLUB. Real-world MI minimization experiments, including domain adaptation and information bottleneck, demonstrate the effectiveness of the proposed method. The code is at https://github.com/Linear95/CLUB.","url_abs":"https://arxiv.org/abs/2006.12013v6","url_pdf":"https://arxiv.org/pdf/2006.12013v6.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"club-a-contrastive-log-ratio-upper-bound-of","repo_url":"https://github.com/Linear95/CLUB","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"tf","reach":null},{"paper_slug":"club-a-contrastive-log-ratio-upper-bound-of","repo_url":"https://github.com/linear95/tc-estimation","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"unanswered"}}],"tasks":[{"task_slug":"domain-adaptation","task_name":"Domain Adaptation"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2006.12013","atlas_url":"https://app.syntology.ai/?focus=2006.12013","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2006.12013"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/Linear95/CLUB","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/linear95/tc-estimation","reach":{"status":"unanswered"}}],"summary":{"ran_draft_wrong":1},"by_repo_kind":{"official":{"samples":1,"ran":1,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":1,"samples":[{"code_sha256_prefix":"a1572564f4e423a2","entry":"log_sum_exp","repo":"Linear95/CLUB","repo_kind":"official","path":"mi_estimators.py","file_url":"https://github.com/Linear95/CLUB/blob/HEAD/mi_estimators.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"a1572564f4e423a2"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}