Papers › MIND: Inductive Mutual Information Estimation, A Convex Maximum-Entropy Copula Approach
MIND: Inductive Mutual Information Estimation, A Convex Maximum-Entropy Copula Approach
Yves-Laurent Kom Samo
We propose a novel estimator of the mutual information between two ordinal vectors x and y. Our approach is inductive (as opposed to deductive) in that it depends on the data generating distribution solely through some nonparametric properties revealing associations in the data, and does not require having enough data to fully characterize the true joint distributions P_(x, y). Specifically, our approach consists of (i) noting that I(y; x) = I(u_y; uₓ) where u_y and uₓ are the copula-uniform dual representations of y and x (i.e. their images under the probability integral transform), and (ii) estimating the copula entropies h(u_y), h(uₓ) and h(u_y, uₓ) by solving a maximum-entropy problem over the space of copula densities under a constraint of the type αₘ = E[ϕₘ(u_y, uₓ)]. We prove that, so long as the constraint is feasible, this problem admits a unique solution, it is in the exponential family, and it can be learned by solving a convex optimization problem. The resulting estimator, which we denote MIND, is marginal-invariant, always non-negative, unbounded for any sample size n, consistent, has MSE rate O(1/n), and is more data-efficient than competing approaches. Beyond mutual information estimation, we illustrate that our approach may be used to mitigate mode collapse in GANs by maximizing the entropy of the copula of fake samples, a model we refer to as Copula Entropy Regularized GAN (CER-GAN).
In Syntology Open this paper in Syntology's Atlas, the map of the papers in Syntology's graph and their citations.
Code
Repository list and official/mentioned flags are the archive's, frozen 2025-07-28. Reachability, where shown, is from one Syntology probe window (2026-09-16 to 2026-09-18); repositories not probed show nothing. GitHub stars are not tracked.
Code Syntology ran Syntology
Not run by Syntology. Nothing on this page verifies that the listed code works.
Tasks
Results from the paper archive 2025-07-28
No leaderboard rows for this paper in the archive.
Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections