{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/arxiv-2510-04280","title":"A KL-regularization Framework for Learning to Plan with Adaptive Priors","arxiv_id":"2510.04280","date":"2025-10-05","proceeding":"ICML","authors":["Álvaro Serra-Gomez","Daniel Jarne Ornia","Dhruva Tirumala","Thomas Moerland"],"abstract":"Effective exploration remains a central challenge in model-based reinforcement learning (MBRL), particularly in high-dimensional continuous control tasks where sample efficiency is crucial. A prominent line of recent work leverages learned policies as proposal distributions for Model-Predictive Path Integral (MPPI) planning. Initial approaches update the sampling policy independently of the planner distribution, typically maximizing a learned value function with deterministic policy gradient and entropy regularization. However, because the states encountered during training depend on the MPPI planner, aligning the sampling policy with the planner improves the accuracy of value estimation and long-term performance. To this end, recent methods update the sampling policy by minimizing KL divergence to the planner distribution or by introducing planner-guided regularization into the policy update. In this work, we unify these MPPI-based reinforcement learning methods under a single framework by introducing Policy Optimization-Model Predictive Control (PO-MPC), a family of KL-regularized MBRL methods that integrate the planner's action distribution as a prior in policy optimization. By aligning the learned policy with the planner's behavior, PO-MPC allows more flexibility in the policy updates to trade off Return maximization and KL divergence minimization. We clarify how prior approaches emerge as special cases of this family, and we explore previously unstudied variations. Our experiments show that these extended configurations yield significant performance improvements, advancing the state of the art in MPPI-based RL.","url_abs":"https://arxiv.org/abs/2510.04280","url_pdf":"https://arxiv.org/pdf/2510.04280","source":{"archive":null,"snapshot":"2025-07-28","note":"not in the Papers with Code archive (frozen at the snapshot)","row_kind":"graph","title_abstract_authors_date":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)"},"code_links":[],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2510.04280","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2510.04280"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"mentioned_in_github":null,"is_official":null,"provenance":"deterministic:regex_extraction","mentioned_in_paper":null,"url":"https://github.com/alvaro-serra/pompc","reach":{"status":"ok"}}],"summary":{"unverified":8},"by_repo_kind":{"found_in_text":{"samples":8,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":8,"samples":[{"code_sha256_prefix":"73a9a088a43ba933","entry":"make_humanoid_env","repo":"alvaro-serra/pompc","repo_kind":"found_in_text","path":"pompc/envs/humanoid.py","file_url":"https://github.com/alvaro-serra/pompc/blob/HEAD/pompc/envs/humanoid.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"73a9a088a43ba933"}},{"code_sha256_prefix":"870d003b125d9510","entry":"mean_std_normalization","repo":"alvaro-serra/pompc","repo_kind":"found_in_text","path":"pompc/common/scale.py","file_url":"https://github.com/alvaro-serra/pompc/blob/HEAD/pompc/common/scale.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"870d003b125d9510"}},{"code_sha256_prefix":"4d597c82f6f0429e","entry":"mish","repo":"alvaro-serra/pompc","repo_kind":"found_in_text","path":"pompc/common/activations.py","file_url":"https://github.com/alvaro-serra/pompc/blob/HEAD/pompc/common/activations.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"4d597c82f6f0429e"}},{"code_sha256_prefix":"4bb5725d10c9a63a","entry":"percentile_normalization","repo":"alvaro-serra/pompc","repo_kind":"found_in_text","path":"pompc/common/scale.py","file_url":"https://github.com/alvaro-serra/pompc/blob/HEAD/pompc/common/scale.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"4bb5725d10c9a63a"}},{"code_sha256_prefix":"67d08c1b698d8370","entry":"simnorm","repo":"alvaro-serra/pompc","repo_kind":"found_in_text","path":"pompc/common/activations.py","file_url":"https://github.com/alvaro-serra/pompc/blob/HEAD/pompc/common/activations.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"67d08c1b698d8370"}},{"code_sha256_prefix":"d241520c8a517f0a","entry":"symexp","repo":"alvaro-serra/pompc","repo_kind":"found_in_text","path":"pompc/common/util.py","file_url":"https://github.com/alvaro-serra/pompc/blob/HEAD/pompc/common/util.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"d241520c8a517f0a"}},{"code_sha256_prefix":"89c9792c5bbea8e1","entry":"symlog","repo":"alvaro-serra/pompc","repo_kind":"found_in_text","path":"pompc/common/util.py","file_url":"https://github.com/alvaro-serra/pompc/blob/HEAD/pompc/common/util.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"89c9792c5bbea8e1"}},{"code_sha256_prefix":"f97a336a0ee5a560","entry":"two_hot","repo":"alvaro-serra/pompc","repo_kind":"found_in_text","path":"pompc/common/util.py","file_url":"https://github.com/alvaro-serra/pompc/blob/HEAD/pompc/common/util.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"f97a336a0ee5a560"}}]},"arxiv_metadata":{"licence":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)","fields":["title","abstract","authors","date"],"primary_category":"cs.LG","source":"arxiv_api"},"syntology_extracted_results":null}