{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/efficient-reinforcement-finetuning-via","title":"Efficient Reinforcement Finetuning via Adaptive Curriculum Learning","arxiv_id":"2504.05520","date":"2025-04-07","proceeding":null,"authors":["Taiwei Shi","Yiyang Wu","Linxin Song","Tianyi Zhou","Jieyu Zhao"],"abstract":"Reinforcement finetuning (RFT) has shown great potential for enhancing the mathematical reasoning capabilities of large language models (LLMs), but it is often sample- and compute-inefficient, requiring extensive training. In this work, we introduce AdaRFT (Adaptive Curriculum Reinforcement Finetuning), a method that significantly improves both the efficiency and final accuracy of RFT through adaptive curriculum learning. AdaRFT dynamically adjusts the difficulty of training problems based on the model's recent reward signals, ensuring that the model consistently trains on tasks that are challenging but solvable. This adaptive sampling strategy accelerates learning by maintaining an optimal difficulty range, avoiding wasted computation on problems that are too easy or too hard. AdaRFT requires only a lightweight extension to standard RFT algorithms like Proximal Policy Optimization (PPO), without modifying the reward function or model architecture. Experiments on competition-level math datasets-including AMC, AIME, and IMO-style problems-demonstrate that AdaRFT significantly improves both training efficiency and reasoning performance. We evaluate AdaRFT across multiple data distributions and model sizes, showing that it reduces training time by up to 2x and improves accuracy by a considerable margin, offering a more scalable and effective RFT framework.","url_abs":"https://arxiv.org/abs/2504.05520v2","url_pdf":"https://arxiv.org/pdf/2504.05520v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"efficient-reinforcement-finetuning-via","repo_url":"https://github.com/uscnlp-lime/verl","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"math","task_name":"Math"},{"task_slug":"mathematical-reasoning","task_name":"Mathematical Reasoning"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2504.05520","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2504.05520"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/uscnlp-lime/verl","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran_draft_wrong":2,"ran_honours":1,"ran":1,"ran_fixture":1,"unverified":10},"by_repo_kind":{"official":{"samples":15,"ran":5,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"f887d72c9492d9ee","entry":"convert_to_regular_types","repo":"uscnlp-lime/verl","repo_kind":"official","path":"verl/trainer/fsdp_sft_trainer.py","file_url":"https://github.com/uscnlp-lime/verl/blob/HEAD/verl/trainer/fsdp_sft_trainer.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"f887d72c9492d9ee"}},{"code_sha256_prefix":"216e28c040173a61","entry":"extract_step","repo":"uscnlp-lime/verl","repo_kind":"official","path":"verl/trainer/fsdp_sft_trainer.py","file_url":"https://github.com/uscnlp-lime/verl/blob/HEAD/verl/trainer/fsdp_sft_trainer.py","link_basis":"harvester_set","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"216e28c040173a61"}},{"code_sha256_prefix":"42a7954a8cfa89e0","entry":"is_transformers_version_in_range","repo":"uscnlp-lime/verl","repo_kind":"official","path":"verl/models/transformers/monkey_patch.py","file_url":"https://github.com/uscnlp-lime/verl/blob/HEAD/verl/models/transformers/monkey_patch.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"42a7954a8cfa89e0"}},{"code_sha256_prefix":"21331a58f93375e4","entry":"union_tensor_dict","repo":"uscnlp-lime/verl","repo_kind":"official","path":"verl/protocol.py","file_url":"https://github.com/uscnlp-lime/verl/blob/HEAD/verl/protocol.py","link_basis":"harvester_set","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"DEP_MISSING","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"21331a58f93375e4"}},{"code_sha256_prefix":"25f0ea3f460f6ce1","entry":"unpad_dataproto","repo":"uscnlp-lime/verl","repo_kind":"official","path":"verl/protocol.py","file_url":"https://github.com/uscnlp-lime/verl/blob/HEAD/verl/protocol.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"25f0ea3f460f6ce1"}},{"code_sha256_prefix":"736144d442549c26","entry":"flash_attention_forward","repo":"uscnlp-lime/verl","repo_kind":"official","path":"verl/models/transformers/qwen2_vl.py","file_url":"https://github.com/uscnlp-lime/verl/blob/HEAD/verl/models/transformers/qwen2_vl.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"736144d442549c26"}},{"code_sha256_prefix":"660713a8fa047736","entry":"get_generation_config","repo":"uscnlp-lime/verl","repo_kind":"official","path":"verl/utils/model.py","file_url":"https://github.com/uscnlp-lime/verl/blob/HEAD/verl/utils/model.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"660713a8fa047736"}},{"code_sha256_prefix":"fe79c459a3e4cc64","entry":"get_huggingface_actor_config","repo":"uscnlp-lime/verl","repo_kind":"official","path":"verl/utils/model.py","file_url":"https://github.com/uscnlp-lime/verl/blob/HEAD/verl/utils/model.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"fe79c459a3e4cc64"}},{"code_sha256_prefix":"e29ba2dcd05add9f","entry":"get_rope_index","repo":"uscnlp-lime/verl","repo_kind":"official","path":"verl/models/transformers/qwen2_vl.py","file_url":"https://github.com/uscnlp-lime/verl/blob/HEAD/verl/models/transformers/qwen2_vl.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"e29ba2dcd05add9f"}},{"code_sha256_prefix":"39fba8c5b01d595e","entry":"get_weight_loader","repo":"uscnlp-lime/verl","repo_kind":"official","path":"verl/models/weight_loader_registry.py","file_url":"https://github.com/uscnlp-lime/verl/blob/HEAD/verl/models/weight_loader_registry.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"39fba8c5b01d595e"}},{"code_sha256_prefix":"8bf1b3e9bf87072c","entry":"get_weight_saver","repo":"uscnlp-lime/verl","repo_kind":"official","path":"verl/models/weight_loader_registry.py","file_url":"https://github.com/uscnlp-lime/verl/blob/HEAD/verl/models/weight_loader_registry.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"8bf1b3e9bf87072c"}},{"code_sha256_prefix":"1901b047ab8e1654","entry":"load_state_dict_to_megatron_gptmodel","repo":"uscnlp-lime/verl","repo_kind":"official","path":"verl/models/mcore/loader.py","file_url":"https://github.com/uscnlp-lime/verl/blob/HEAD/verl/models/mcore/loader.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"1901b047ab8e1654"}},{"code_sha256_prefix":"e89bd73550a8c08f","entry":"prepare_fa2_from_position_ids","repo":"uscnlp-lime/verl","repo_kind":"official","path":"verl/models/transformers/qwen2_vl.py","file_url":"https://github.com/uscnlp-lime/verl/blob/HEAD/verl/models/transformers/qwen2_vl.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"e89bd73550a8c08f"}},{"code_sha256_prefix":"00f0f9848c8cd16c","entry":"repeat_kv","repo":"uscnlp-lime/verl","repo_kind":"official","path":"verl/models/transformers/monkey_patch.py","file_url":"https://github.com/uscnlp-lime/verl/blob/HEAD/verl/models/transformers/monkey_patch.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"00f0f9848c8cd16c"}},{"code_sha256_prefix":"3b15e2ac7497c441","entry":"squeeze","repo":"uscnlp-lime/verl","repo_kind":"official","path":"verl/utils/model.py","file_url":"https://github.com/uscnlp-lime/verl/blob/HEAD/verl/utils/model.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"3b15e2ac7497c441"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}