{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/hyperspherical-normalization-for-scalable","title":"Hyperspherical Normalization for Scalable Deep Reinforcement Learning","arxiv_id":"2502.15280","date":"2025-02-21","proceeding":null,"authors":["Hojoon Lee","Youngdo Lee","Takuma Seno","Donghu Kim","Peter Stone","Jaegul Choo"],"abstract":"Scaling up the model size and computation has brought consistent performance improvements in supervised learning. However, this lesson often fails to apply to reinforcement learning (RL) because training the model on non-stationary data easily leads to overfitting and unstable optimization. In response, we introduce SimbaV2, a novel RL architecture designed to stabilize optimization by (i) constraining the growth of weight and feature norm by hyperspherical normalization; and (ii) using a distributional value estimation with reward scaling to maintain stable gradients under varying reward magnitudes. Using the soft actor-critic as a base algorithm, SimbaV2 scales up effectively with larger models and greater compute, achieving state-of-the-art performance on 57 continuous control tasks across 4 domains. The code is available at https://dojeon-ai.github.io/SimbaV2.","url_abs":"https://arxiv.org/abs/2502.15280v1","url_pdf":"https://arxiv.org/pdf/2502.15280v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[],"tasks":[{"task_slug":"continuous-control","task_name":"Continuous Control"},{"task_slug":"deep-reinforcement-learning","task_name":"Deep Reinforcement Learning"},{"task_slug":"reinforcement-learning","task_name":"Reinforcement Learning"},{"task_slug":"reinforcement-learning-1","task_name":"Reinforcement Learning (RL)"},{"task_slug":"continuous-control","task_name":"continuous-control"},{"task_slug":"reinforcement-learning-2","task_name":"reinforcement-learning"}],"methods":[{"method_slug":"base","method_name":"BASE"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2502.15280","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2502.15280"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/naumix/BiggerRegularizedOptimistic","reach":null},{"provenance":"deterministic:regex_extraction","url":"https://github.com/adityab/CrossQ","reach":{"status":"ok","spdx":"NOASSERTION"}},{"provenance":"deterministic:regex_extraction","url":"https://github.com/facebookresearch/MRQ","reach":null},{"provenance":"deterministic:regex_extraction","url":"https://github.com/SonyResearch/simba","reach":null}],"summary":{"ran":2,"ran_draft_wrong":1,"ran_honours":1,"unverified":2},"by_repo_kind":{"found_in_text":{"samples":6,"ran":4,"repositories":2}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"f73fa4e26e899f5b","entry":"Model","repo":"naumix/BiggerRegularizedOptimistic","repo_kind":"found_in_text","path":"jaxrl/bro/actor.py","file_url":"https://github.com/naumix/BiggerRegularizedOptimistic/blob/HEAD/jaxrl/bro/actor.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f73fa4e26e899f5b"}},{"code_sha256_prefix":"95538ae49008d867","entry":"SaveState","repo":"naumix/BiggerRegularizedOptimistic","repo_kind":"found_in_text","path":"jaxrl/bro/actor.py","file_url":"https://github.com/naumix/BiggerRegularizedOptimistic/blob/HEAD/jaxrl/bro/actor.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"95538ae49008d867"}},{"code_sha256_prefix":"5ac49b3249fc3ab0","entry":"orthogonal_init","repo":"SonyResearch/simba","repo_kind":"found_in_text","path":"scale_rl/networks/layers.py","file_url":"https://github.com/SonyResearch/simba/blob/HEAD/scale_rl/networks/layers.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"5ac49b3249fc3ab0"}},{"code_sha256_prefix":"f325964f1797f651","entry":"tree_norm","repo":"naumix/BiggerRegularizedOptimistic","repo_kind":"found_in_text","path":"jaxrl/bro/actor.py","file_url":"https://github.com/naumix/BiggerRegularizedOptimistic/blob/HEAD/jaxrl/bro/actor.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f325964f1797f651"}},{"code_sha256_prefix":"6b074234d267bb66","entry":"MLPBlock","repo":"SonyResearch/simba","repo_kind":"found_in_text","path":"scale_rl/networks/layers.py","file_url":"https://github.com/SonyResearch/simba/blob/HEAD/scale_rl/networks/layers.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"6b074234d267bb66"}},{"code_sha256_prefix":"9cc22255440aae4f","entry":"update_optimistic","repo":"naumix/BiggerRegularizedOptimistic","repo_kind":"found_in_text","path":"jaxrl/bro/actor.py","file_url":"https://github.com/naumix/BiggerRegularizedOptimistic/blob/HEAD/jaxrl/bro/actor.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"9cc22255440aae4f"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}