{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/model-based-micro-data-reinforcement-learning-1","title":"Model-based micro-data reinforcement learning: what are the crucial model properties and which model to choose?","arxiv_id":"2107.11587","date":"2021-07-24","proceeding":"ICLR 2021 1","authors":["Balázs Kégl","Gabriel Hurtado","Albert Thomas"],"abstract":"We contribute to micro-data model-based reinforcement learning (MBRL) by rigorously comparing popular generative models using a fixed (random shooting) control agent. We find that on an environment that requires multimodal posterior predictives, mixture density nets outperform all other models by a large margin. When multimodality is not required, our surprising finding is that we do not need probabilistic posterior predictives: deterministic models are on par, in fact they consistently (although non-significantly) outperform their probabilistic counterparts. We also found that heteroscedasticity at training time, perhaps acting as a regularizer, improves predictions at longer horizons. At the methodological side, we design metrics and an experimental protocol which can be used to evaluate the various models, predicting their asymptotic performance when using them on the control problem. Using this framework, we improve the state-of-the-art sample complexity of MBRL on Acrobot by two to four folds, using an aggressive training schedule which is outside of the hyperparameter interval usually considered","url_abs":"https://arxiv.org/abs/2107.11587v1","url_pdf":"https://arxiv.org/pdf/2107.11587v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"model-based-micro-data-reinforcement-learning-1","repo_url":"https://github.com/ramp-kits/rl_simulator","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok","spdx":"BSD-3-Clause"}}],"tasks":[{"task_slug":"acrobot","task_name":"Acrobot"},{"task_slug":"model-based-reinforcement-learning","task_name":"Model-based Reinforcement Learning"},{"task_slug":"model","task_name":"model"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2107.11587","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2107.11587"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/ramp-kits/rl_simulator","reach":{"status":"ok","spdx":"BSD-3-Clause"}}],"summary":{"unverified":2},"by_repo_kind":{"official":{"samples":2,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":2,"samples":[{"code_sha256_prefix":"e6746c13a6675e9d","entry":"mbrl_hyperopt_command","repo":"ramp-kits/rl_simulator","repo_kind":"official","path":"mbrl-tools/mbrltools/model_based_rl_hyperopt.py","file_url":"https://github.com/ramp-kits/rl_simulator/blob/HEAD/mbrl-tools/mbrltools/model_based_rl_hyperopt.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":false,"mcp_get_code":{"code_sha256":"e6746c13a6675e9d"}},{"code_sha256_prefix":"d5e9182e08dfe611","entry":"reward_func","repo":"ramp-kits/rl_simulator","repo_kind":"official","path":"benchmark/acrobot/reward_function.py","file_url":"https://github.com/ramp-kits/rl_simulator/blob/HEAD/benchmark/acrobot/reward_function.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":false,"mcp_get_code":{"code_sha256":"d5e9182e08dfe611"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}