{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/remedi-corrective-transformations-for","title":"REMEDI: Corrective Transformations for Improved Neural Entropy Estimation","arxiv_id":"2402.05718","date":"2024-02-08","proceeding":null,"authors":["Viktor Nilsson","Anirban Samaddar","Sandeep Madireddy","Pierre Nyquist"],"abstract":"Information theoretic quantities play a central role in machine learning. The recent surge in the complexity of data and models has increased the demand for accurate estimation of these quantities. However, as the dimension grows the estimation presents significant challenges, with existing methods struggling already in relatively low dimensions. To address this issue, in this work, we introduce $\\texttt{REMEDI}$ for efficient and accurate estimation of differential entropy, a fundamental information theoretic quantity. The approach combines the minimization of the cross-entropy for simple, adaptive base models and the estimation of their deviation, in terms of the relative entropy, from the data density. Our approach demonstrates improvement across a broad spectrum of estimation tasks, encompassing entropy estimation on both synthetic and natural data. Further, we extend important theoretical consistency results to a more generalized setting required by our approach. We illustrate how the framework can be naturally extended to information theoretic supervised learning models, with a specific focus on the Information Bottleneck approach. It is demonstrated that the method delivers better accuracy compared to the existing methods in Information Bottleneck. In addition, we explore a natural connection between $\\texttt{REMEDI}$ and generative modeling using rejection sampling and Langevin dynamics.","url_abs":"https://arxiv.org/abs/2402.05718v2","url_pdf":"https://arxiv.org/pdf/2402.05718v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"remedi-corrective-transformations-for","repo_url":"https://github.com/viktor765/remedi","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":null}],"tasks":[],"methods":[{"method_slug":"base","method_name":"BASE"},{"method_slug":"focus","method_name":"Focus"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2402.05718","atlas_url":"https://app.syntology.ai/?focus=2402.05718","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2402.05718"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/viktor765/remedi","reach":null}],"summary":{"ran":2,"unverified":1},"by_repo_kind":{"official":{"samples":3,"ran":2,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":3,"samples":[{"code_sha256_prefix":"647bafcbd560b2b7","entry":"KNIFE","repo":"viktor765/remedi","repo_kind":"official","path":"Information bottleneck/REMEDI_MLP.py","file_url":"https://github.com/viktor765/remedi/blob/HEAD/Information%20bottleneck/REMEDI_MLP.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"647bafcbd560b2b7"}},{"code_sha256_prefix":"615c65c649bebb14","entry":"MargKernel","repo":"viktor765/remedi","repo_kind":"official","path":"Information bottleneck/REMEDI_MLP.py","file_url":"https://github.com/viktor765/remedi/blob/HEAD/Information%20bottleneck/REMEDI_MLP.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"615c65c649bebb14"}},{"code_sha256_prefix":"0a6854074eb182e8","entry":"REMEDI","repo":"viktor765/remedi","repo_kind":"official","path":"Information bottleneck/REMEDI_MLP.py","file_url":"https://github.com/viktor765/remedi/blob/HEAD/Information%20bottleneck/REMEDI_MLP.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"0a6854074eb182e8"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}