{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/estimation-of-kl-divergence-optimal-minimax","title":"Estimation of KL Divergence: Optimal Minimax Rate","arxiv_id":"1607.02653","date":"2016-07-09","proceeding":null,"authors":["Yuheng Bu","Shaofeng Zou","Yingbin Liang","Venugopal V. Veeravalli"],"abstract":"The problem of estimating the Kullback-Leibler divergence $D(P\\|Q)$ between two unknown distributions $P$ and $Q$ is studied, under the assumption that the alphabet size $k$ of the distributions can scale to infinity. The estimation is based on $m$ independent samples drawn from $P$ and $n$ independent samples drawn from $Q$. It is first shown that there does not exist any consistent estimator that guarantees asymptotically small worst-case quadratic risk over the set of all pairs of distributions. A restricted set that contains pairs of distributions, with density ratio bounded by a function $f(k)$ is further considered. {An augmented plug-in estimator is proposed, and its worst-case quadratic risk is shown to be within a constant factor of $(\\frac{k}{m}+\\frac{kf(k)}{n})^2+\\frac{\\log ^2 f(k)}{m}+\\frac{f(k)}{n}$, if $m$ and $n$ exceed a constant factor of $k$ and $kf(k)$, respectively.} Moreover, the minimax quadratic risk is characterized to be within a constant factor of $(\\frac{k}{m\\log k}+\\frac{kf(k)}{n\\log k})^2+\\frac{\\log ^2 f(k)}{m}+\\frac{f(k)}{n}$, if $m$ and $n$ exceed a constant factor of $k/\\log(k)$ and $kf(k)/\\log k$, respectively. The lower bound on the minimax quadratic risk is characterized by employing a generalized Le Cam's method. A minimax optimal estimator is then constructed by employing both the polynomial approximation and the plug-in approaches.","url_abs":"http://arxiv.org/abs/1607.02653v4","url_pdf":"http://arxiv.org/pdf/1607.02653v4.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"links_only","authors_date_abstract":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license), from the Kaggle arXiv metadata snapshot of 2026-09-12"},"code_links":[{"paper_slug":"estimation-of-kl-divergence-optimal-minimax","repo_url":"https://github.com/buyuheng/Minimax-KL-divergence-estimator","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"none","reach":{"status":"ok"}}],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=1607.02653","mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}