{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/better-estimation-of-the-kl-divergence","title":"Better Estimation of the KL Divergence Between Language Models","arxiv_id":"2504.10637","date":"2025-04-14","proceeding":null,"authors":["Afra Amini","Tim Vieira","Ryan Cotterell"],"abstract":"Estimating the Kullback--Leibler (KL) divergence between language models has many applications, e.g., reinforcement learning from human feedback (RLHF), interpretability, and knowledge distillation. However, computing the exact KL divergence between two arbitrary language models is intractable. Thus, practitioners often resort to the use of sampling-based estimators. While it is easy to fashion a simple Monte Carlo (MC) estimator that provides an unbiased estimate of the KL divergence between language models, this estimator notoriously suffers from high variance, and can even result in a negative estimate of the KL divergence, a non-negative quantity. In this paper, we introduce a Rao--Blackwellized estimator that is also unbiased and provably has variance less than or equal to that of the standard Monte Carlo estimator. In an empirical study on sentiment-controlled fine-tuning, we show that our estimator provides more stable KL estimates and reduces variance substantially in practice. Additionally, we derive an analogous Rao--Blackwellized estimator of the gradient of the KL divergence, which leads to more stable training and produces models that more frequently appear on the Pareto frontier of reward vs. KL compared to the ones trained with the MC estimator of the gradient.","url_abs":"https://arxiv.org/abs/2504.10637v2","url_pdf":"https://arxiv.org/pdf/2504.10637v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"better-estimation-of-the-kl-divergence","repo_url":"https://github.com/rycolab/kl-rb","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"knowledge-distillation","task_name":"Knowledge Distillation"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2504.10637","atlas_url":"https://app.syntology.ai/?focus=2504.10637","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2504.10637"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/rycolab/kl-rb","reach":null}],"summary":{"ran_honours":3,"ran_draft_wrong":2},"by_repo_kind":{"official":{"samples":5,"ran":5,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":5,"samples":[{"code_sha256_prefix":"2fee0ec1e42db0e2","entry":"pad","repo":"rycolab/kl-rb","repo_kind":"official","path":"trl/trainer/utils.py","file_url":"https://github.com/rycolab/kl-rb/blob/HEAD/trl/trainer/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":2,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"2fee0ec1e42db0e2"}},{"code_sha256_prefix":"fd37f7ac629f7673","entry":"compute_kl","repo":"rycolab/kl-rb","repo_kind":"official","path":"trl/trainer/utils.py","file_url":"https://github.com/rycolab/kl-rb/blob/HEAD/trl/trainer/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"fd37f7ac629f7673"}},{"code_sha256_prefix":"2887fe30b3acdea7","entry":"get_global_statistics","repo":"rycolab/kl-rb","repo_kind":"official","path":"trl/trainer/utils.py","file_url":"https://github.com/rycolab/kl-rb/blob/HEAD/trl/trainer/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"2887fe30b3acdea7"}},{"code_sha256_prefix":"b6aad876161ad697","entry":"get_positive_score","repo":"rycolab/kl-rb","repo_kind":"official","path":"rloo_sentiment.py","file_url":"https://github.com/rycolab/kl-rb/blob/HEAD/rloo_sentiment.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"b6aad876161ad697"}},{"code_sha256_prefix":"8c77fc1b61189419","entry":"load_prompts","repo":"rycolab/kl-rb","repo_kind":"official","path":"rloo_sentiment.py","file_url":"https://github.com/rycolab/kl-rb/blob/HEAD/rloo_sentiment.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"8c77fc1b61189419"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}