{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/discrete-codebook-world-models-for-continuous","title":"Discrete Codebook World Models for Continuous Control","arxiv_id":"2503.00653","date":"2025-03-01","proceeding":null,"authors":["Aidan Scannell","Mohammadreza Nakhaei","Kalle Kujanpää","Yi Zhao","Kevin Sebastian Luck","Arno Solin","Joni Pajarinen"],"abstract":"In reinforcement learning (RL), world models serve as internal simulators, enabling agents to predict environment dynamics and future outcomes in order to make informed decisions. While previous approaches leveraging discrete latent spaces, such as DreamerV3, have demonstrated strong performance in discrete action settings and visual control tasks, their comparative performance in state-based continuous control remains underexplored. In contrast, methods with continuous latent spaces, such as TD-MPC2, have shown notable success in state-based continuous control benchmarks. In this paper, we demonstrate that modeling discrete latent states has benefits over continuous latent states and that discrete codebook encodings are more effective representations for continuous control, compared to alternative encodings, such as one-hot and label-based encodings. Based on these insights, we introduce DCWM: Discrete Codebook World Model, a self-supervised world model with a discrete and stochastic latent space, where latent states are codes from a codebook. We combine DCWM with decision-time planning to get our model-based RL algorithm, named DC-MPC: Discrete Codebook Model Predictive Control, which performs competitively against recent state-of-the-art algorithms, including TD-MPC2 and DreamerV3, on continuous control benchmarks. See our project website www.aidanscannell.com/dcmpc.","url_abs":"https://arxiv.org/abs/2503.00653v1","url_pdf":"https://arxiv.org/pdf/2503.00653v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"discrete-codebook-world-models-for-continuous","repo_url":"https://github.com/aidanscannell/dcmpc","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"continuous-control","task_name":"Continuous Control"},{"task_slug":"model-predictive-control","task_name":"Model Predictive Control"},{"task_slug":"reinforcement-learning-1","task_name":"Reinforcement Learning (RL)"},{"task_slug":"continuous-control","task_name":"continuous-control"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2503.00653","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2503.00653"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/aidanscannell/dcmpc","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"unverified":8},"by_repo_kind":{"official":{"samples":8,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"f1acb525c0929888","entry":"calc_mean_opt_moments","repo":"aidanscannell/dcmpc","repo_kind":"official","path":"utils/helper.py","file_url":"https://github.com/aidanscannell/dcmpc/blob/HEAD/utils/helper.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f1acb525c0929888"}},{"code_sha256_prefix":"238266345b37c6dc","entry":"calc_rank","repo":"aidanscannell/dcmpc","repo_kind":"official","path":"utils/helper.py","file_url":"https://github.com/aidanscannell/dcmpc/blob/HEAD/utils/helper.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"238266345b37c6dc"}},{"code_sha256_prefix":"1d3835836ba9076b","entry":"evaluate","repo":"aidanscannell/dcmpc","repo_kind":"official","path":"utils/evaluate.py","file_url":"https://github.com/aidanscannell/dcmpc/blob/HEAD/utils/evaluate.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"1d3835836ba9076b"}},{"code_sha256_prefix":"5463329fd15b6ca9","entry":"flatten_batch","repo":"aidanscannell/dcmpc","repo_kind":"official","path":"utils/buffers.py","file_url":"https://github.com/aidanscannell/dcmpc/blob/HEAD/utils/buffers.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"5463329fd15b6ca9"}},{"code_sha256_prefix":"9acd81aa05192bdb","entry":"mlp","repo":"aidanscannell/dcmpc","repo_kind":"official","path":"utils/layers.py","file_url":"https://github.com/aidanscannell/dcmpc/blob/HEAD/utils/layers.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"9acd81aa05192bdb"}},{"code_sha256_prefix":"6c36e9ed51bed5c6","entry":"mlp_ensemble","repo":"aidanscannell/dcmpc","repo_kind":"official","path":"utils/layers.py","file_url":"https://github.com/aidanscannell/dcmpc/blob/HEAD/utils/layers.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"6c36e9ed51bed5c6"}},{"code_sha256_prefix":"7fb5eb47c3b273d5","entry":"symlog","repo":"aidanscannell/dcmpc","repo_kind":"official","path":"utils/helper.py","file_url":"https://github.com/aidanscannell/dcmpc/blob/HEAD/utils/helper.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"7fb5eb47c3b273d5"}},{"code_sha256_prefix":"9babf91705bd55e7","entry":"to_nstep","repo":"aidanscannell/dcmpc","repo_kind":"official","path":"utils/buffers.py","file_url":"https://github.com/aidanscannell/dcmpc/blob/HEAD/utils/buffers.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"9babf91705bd55e7"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}