{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/hyperparameters-in-reinforcement-learning-and","title":"Hyperparameters in Reinforcement Learning and How To Tune Them","arxiv_id":"2306.01324","date":"2023-06-02","proceeding":null,"authors":["Theresa Eimer","Marius Lindauer","Roberta Raileanu"],"abstract":"In order to improve reproducibility, deep reinforcement learning (RL) has been adopting better scientific practices such as standardized evaluation metrics and reporting. However, the process of hyperparameter optimization still varies widely across papers, which makes it challenging to compare RL algorithms fairly. In this paper, we show that hyperparameter choices in RL can significantly affect the agent's final performance and sample efficiency, and that the hyperparameter landscape can strongly depend on the tuning seed which may lead to overfitting. We therefore propose adopting established best practices from AutoML, such as the separation of tuning and testing seeds, as well as principled hyperparameter optimization (HPO) across a broad search space. We support this by comparing multiple state-of-the-art HPO tools on a range of RL algorithms and environments to their hand-tuned counterparts, demonstrating that HPO approaches often have higher performance and lower compute overhead. As a result of our findings, we recommend a set of best practices for the RL community, which should result in stronger empirical results with fewer computational costs, better reproducibility, and thus faster progress. In order to encourage the adoption of these practices, we provide plug-and-play implementations of the tuning algorithms used in this paper at https://github.com/facebookresearch/how-to-autorl.","url_abs":"https://arxiv.org/abs/2306.01324v1","url_pdf":"https://arxiv.org/pdf/2306.01324v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"hyperparameters-in-reinforcement-learning-and","repo_url":"https://github.com/facebookresearch/how-to-autorl","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"none","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"automl","task_name":"AutoML"},{"task_slug":"deep-reinforcement-learning","task_name":"Deep Reinforcement Learning"},{"task_slug":"hyperparameter-optimization","task_name":"Hyperparameter Optimization"},{"task_slug":"reinforcement-learning","task_name":"Reinforcement Learning"},{"task_slug":"reinforcement-learning-1","task_name":"Reinforcement Learning (RL)"},{"task_slug":"reinforcement-learning-2","task_name":"reinforcement-learning"}],"methods":[{"method_slug":"hpo","method_name":"HPO"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2306.01324","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2306.01324"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/facebookresearch/how-to-autorl","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran_violates":1,"unverified":4},"by_repo_kind":{"official":{"samples":5,"ran":1,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"59d38e267656d07b","entry":"order_stats","repo":"facebookresearch/how-to-autorl","repo_kind":"official","path":"hydra_plugins/hydra_pbt_sweeper/bgt_utils.py","file_url":"https://github.com/facebookresearch/how-to-autorl/blob/HEAD/hydra_plugins/hydra_pbt_sweeper/bgt_utils.py","link_basis":"plan_row","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"59d38e267656d07b"}},{"code_sha256_prefix":"62266f87936c1a59","entry":"UCB","repo":"facebookresearch/how-to-autorl","repo_kind":"official","path":"hydra_plugins/hydra_pbt_sweeper/pb2_utils.py","file_url":"https://github.com/facebookresearch/how-to-autorl/blob/HEAD/hydra_plugins/hydra_pbt_sweeper/pb2_utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"62266f87936c1a59"}},{"code_sha256_prefix":"03eb6648693612dc","entry":"copula_standardize","repo":"facebookresearch/how-to-autorl","repo_kind":"official","path":"hydra_plugins/hydra_pbt_sweeper/bgt_utils.py","file_url":"https://github.com/facebookresearch/how-to-autorl/blob/HEAD/hydra_plugins/hydra_pbt_sweeper/bgt_utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"03eb6648693612dc"}},{"code_sha256_prefix":"e62be274f5b1b070","entry":"normalize","repo":"facebookresearch/how-to-autorl","repo_kind":"official","path":"hydra_plugins/hydra_pbt_sweeper/bgt_utils.py","file_url":"https://github.com/facebookresearch/how-to-autorl/blob/HEAD/hydra_plugins/hydra_pbt_sweeper/bgt_utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"e62be274f5b1b070"}},{"code_sha256_prefix":"c688c3623804842a","entry":"standardize","repo":"facebookresearch/how-to-autorl","repo_kind":"official","path":"hydra_plugins/hydra_pbt_sweeper/pb2_utils.py","file_url":"https://github.com/facebookresearch/how-to-autorl/blob/HEAD/hydra_plugins/hydra_pbt_sweeper/pb2_utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"c688c3623804842a"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}