{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/areal-a-large-scale-asynchronous","title":"AReaL: A Large-Scale Asynchronous Reinforcement Learning System for Language Reasoning","arxiv_id":"2505.24298","date":"2025-05-30","proceeding":null,"authors":["Wei Fu","Jiaxuan Gao","Xujie Shen","Chen Zhu","Zhiyu Mei","Chuyi He","Shusheng Xu","Guo Wei","Jun Mei","Jiashu Wang","Tongkai Yang","Binhang Yuan","Yi Wu"],"abstract":"Reinforcement learning (RL) has become a dominant paradigm for training large language models (LLMs), particularly for reasoning tasks. Effective RL for LLMs requires massive parallelization and poses an urgent need for efficient training systems. Most existing large-scale RL systems for LLMs are synchronous, alternating generation and training in a batch setting where rollouts in each training batch are generated by the same model. This approach stabilizes RL training but suffers from severe system-level inefficiency: generation must wait until the longest output in the batch is completed before model updates, resulting in GPU underutilization. We present AReaL, a fully asynchronous RL system that completely decouples generation from training. Rollout workers in AReaL continuously generate new outputs without waiting, while training workers update the model whenever a batch of data is collected. AReaL also incorporates a collection of system-level optimizations, leading to substantially higher GPU utilization. To stabilize RL training, AReaL balances the workload of rollout and training workers to control data staleness, and adopts a staleness-enhanced PPO variant to better handle outdated training samples. Extensive experiments on math and code reasoning benchmarks show that AReaL achieves up to 2.77$\\times$ training speedup compared to synchronous systems with the same number of GPUs and matched or improved final performance. The code of AReaL is available at https://github.com/inclusionAI/AReaL/.","url_abs":"https://arxiv.org/abs/2505.24298v2","url_pdf":"https://arxiv.org/pdf/2505.24298v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"areal-a-large-scale-asynchronous","repo_url":"https://github.com/inclusionai/areal","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":null,"task_name":"GPU"},{"task_slug":"math","task_name":"Math"},{"task_slug":"reinforcement-learning-1","task_name":"Reinforcement Learning (RL)"}],"methods":[{"method_slug":"entropy-regularization","method_name":"Entropy Regularization"},{"method_slug":"ppo","method_name":"PPO"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2505.24298","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2505.24298"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/inclusionai/areal","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"unverified":13},"by_repo_kind":{"official":{"samples":13,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"984b7305e1675001","entry":"convert_image","repo":"inclusionai/areal","repo_kind":"official","path":"areal/dataset/clevr_count_70k.py","file_url":"https://github.com/inclusionai/areal/blob/HEAD/areal/dataset/clevr_count_70k.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"984b7305e1675001"}},{"code_sha256_prefix":"74fdadc218fb8f82","entry":"convert_image","repo":"inclusionai/areal","repo_kind":"official","path":"areal/dataset/geometry3k.py","file_url":"https://github.com/inclusionai/areal/blob/HEAD/areal/dataset/geometry3k.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"74fdadc218fb8f82"}},{"code_sha256_prefix":"b118bfd2216dcd9b","entry":"detect_image_mime","repo":"inclusionai/areal","repo_kind":"official","path":"areal/api/io_struct.py","file_url":"https://github.com/inclusionai/areal/blob/HEAD/areal/api/io_struct.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"b118bfd2216dcd9b"}},{"code_sha256_prefix":"65a8c4846526b3db","entry":"format_host_for_url","repo":"inclusionai/areal","repo_kind":"official","path":"areal/utils/network.py","file_url":"https://github.com/inclusionai/areal/blob/HEAD/areal/utils/network.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"65a8c4846526b3db"}},{"code_sha256_prefix":"a638861750ab4d98","entry":"format_hostport","repo":"inclusionai/areal","repo_kind":"official","path":"areal/utils/network.py","file_url":"https://github.com/inclusionai/areal/blob/HEAD/areal/utils/network.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"a638861750ab4d98"}},{"code_sha256_prefix":"b882a485cc24570c","entry":"get_gsm8k_rl_dataset","repo":"inclusionai/areal","repo_kind":"official","path":"areal/dataset/gsm8k.py","file_url":"https://github.com/inclusionai/areal/blob/HEAD/areal/dataset/gsm8k.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"b882a485cc24570c"}},{"code_sha256_prefix":"5a0101a5d065cefe","entry":"get_gsm8k_sft_dataset","repo":"inclusionai/areal","repo_kind":"official","path":"areal/dataset/gsm8k.py","file_url":"https://github.com/inclusionai/areal/blob/HEAD/areal/dataset/gsm8k.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"5a0101a5d065cefe"}},{"code_sha256_prefix":"5c929b9172cee02b","entry":"get_py_cmd","repo":"inclusionai/areal","repo_kind":"official","path":"areal/api/cli_args.py","file_url":"https://github.com/inclusionai/areal/blob/HEAD/areal/api/cli_args.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"5c929b9172cee02b"}},{"code_sha256_prefix":"ba5bcb5389640745","entry":"get_versioned_lora_name","repo":"inclusionai/areal","repo_kind":"official","path":"areal/api/io_struct.py","file_url":"https://github.com/inclusionai/areal/blob/HEAD/areal/api/io_struct.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"ba5bcb5389640745"}},{"code_sha256_prefix":"5425dac746421dc0","entry":"gethostip","repo":"inclusionai/areal","repo_kind":"official","path":"areal/utils/network.py","file_url":"https://github.com/inclusionai/areal/blob/HEAD/areal/utils/network.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"5425dac746421dc0"}},{"code_sha256_prefix":"57fa5d6aba08d9d5","entry":"pad_to_square","repo":"inclusionai/areal","repo_kind":"official","path":"areal/dataset/geometry3k.py","file_url":"https://github.com/inclusionai/areal/blob/HEAD/areal/dataset/geometry3k.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"57fa5d6aba08d9d5"}},{"code_sha256_prefix":"f1162ad51bf8451e","entry":"parse_cli_args","repo":"inclusionai/areal","repo_kind":"official","path":"areal/api/cli_args.py","file_url":"https://github.com/inclusionai/areal/blob/HEAD/areal/api/cli_args.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"f1162ad51bf8451e"}},{"code_sha256_prefix":"e0c4f88b3f11687c","entry":"to_structured_cfg","repo":"inclusionai/areal","repo_kind":"official","path":"areal/api/cli_args.py","file_url":"https://github.com/inclusionai/areal/blob/HEAD/areal/api/cli_args.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"e0c4f88b3f11687c"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}