{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/constrained-latent-action-policies-for-model","title":"Constrained Latent Action Policies for Model-Based Offline Reinforcement Learning","arxiv_id":"2411.04562","date":"2024-11-07","proceeding":null,"authors":["Marvin Alles","Philip Becker-Ehmck","Patrick van der Smagt","Maximilian Karl"],"abstract":"In offline reinforcement learning, a policy is learned using a static dataset in the absence of costly feedback from the environment. In contrast to the online setting, only using static datasets poses additional challenges, such as policies generating out-of-distribution samples. Model-based offline reinforcement learning methods try to overcome these by learning a model of the underlying dynamics of the environment and using it to guide policy search. It is beneficial but, with limited datasets, errors in the model and the issue of value overestimation among out-of-distribution states can worsen performance. Current model-based methods apply some notion of conservatism to the Bellman update, often implemented using uncertainty estimation derived from model ensembles. In this paper, we propose Constrained Latent Action Policies (C-LAP) which learns a generative model of the joint distribution of observations and actions. We cast policy learning as a constrained objective to always stay within the support of the latent action distribution, and use the generative capabilities of the model to impose an implicit constraint on the generated actions. Thereby eliminating the need to use additional uncertainty penalties on the Bellman update and significantly decreasing the number of gradient steps required to learn a policy. We empirically evaluate C-LAP on the D4RL and V-D4RL benchmark, and show that C-LAP is competitive to state-of-the-art methods, especially outperforming on datasets with visual observations.","url_abs":"https://arxiv.org/abs/2411.04562v2","url_pdf":"https://arxiv.org/pdf/2411.04562v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"constrained-latent-action-policies-for-model","repo_url":"https://github.com/marvinalles/c-lap","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"jax","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"d4rl","task_name":"D4RL"},{"task_slug":"reinforcement-learning","task_name":"Reinforcement Learning"},{"task_slug":"reinforcement-learning-2","task_name":"reinforcement-learning"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2411.04562","atlas_url":"https://app.syntology.ai/?focus=2411.04562","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2411.04562"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/marvinalles/c-lap","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"unverified":4},"by_repo_kind":{"official":{"samples":4,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"08bbb52a38ebd393","entry":"bernoulli_diag","repo":"marvinalles/c-lap","repo_kind":"official","path":"rlab_jax/distributions.py","file_url":"https://github.com/marvinalles/c-lap/blob/HEAD/rlab_jax/distributions.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"08bbb52a38ebd393"}},{"code_sha256_prefix":"5ca2fd0c3f022a7d","entry":"load_episodes","repo":"marvinalles/c-lap","repo_kind":"official","path":"vd4rl_npz_to_hdf5.py","file_url":"https://github.com/marvinalles/c-lap/blob/HEAD/vd4rl_npz_to_hdf5.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"5ca2fd0c3f022a7d"}},{"code_sha256_prefix":"e3b155f1eaad6674","entry":"multivariate_normal_diag","repo":"marvinalles/c-lap","repo_kind":"official","path":"rlab_jax/distributions.py","file_url":"https://github.com/marvinalles/c-lap/blob/HEAD/rlab_jax/distributions.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e3b155f1eaad6674"}},{"code_sha256_prefix":"a416657c82bf2705","entry":"tanh_multivariate_normal_diag","repo":"marvinalles/c-lap","repo_kind":"official","path":"rlab_jax/distributions.py","file_url":"https://github.com/marvinalles/c-lap/blob/HEAD/rlab_jax/distributions.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a416657c82bf2705"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}