{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/incentivizing-reasoning-for-advanced","title":"Incentivizing Reasoning for Advanced Instruction-Following of Large Language Models","arxiv_id":"2506.01413","date":"2025-06-02","proceeding":null,"authors":["Yulei Qin","Gang Li","Zongyi Li","Zihan Xu","Yuchen Shi","Zhekai Lin","Xiao Cui","Ke Li","Xing Sun"],"abstract":"Existing large language models (LLMs) face challenges of following complex instructions, especially when multiple constraints are present and organized in paralleling, chaining, and branching structures. One intuitive solution, namely chain-of-thought (CoT), is expected to universally improve capabilities of LLMs. However, we find that the vanilla CoT exerts a negative impact on performance due to its superficial reasoning pattern of simply paraphrasing the instructions. It fails to peel back the compositions of constraints for identifying their relationship across hierarchies of types and dimensions. To this end, we propose a systematic method to boost LLMs in dealing with complex instructions via incentivizing reasoning for test-time compute scaling. First, we stem from the decomposition of complex instructions under existing taxonomies and propose a reproducible data acquisition method. Second, we exploit reinforcement learning (RL) with verifiable rule-centric reward signals to cultivate reasoning specifically for instruction following. We address the shallow, non-essential nature of reasoning under complex instructions via sample-wise contrast for superior CoT enforcement. We also exploit behavior cloning of experts to facilitate steady distribution shift from fast-thinking LLMs to skillful reasoners. Extensive evaluations on seven comprehensive benchmarks confirm the validity of the proposed method, where a 1.5B LLM achieves 11.74% gains with performance comparable to a 8B LLM. Codes and data are available at https://github.com/yuleiqin/RAIF.","url_abs":"https://arxiv.org/abs/2506.01413v2","url_pdf":"https://arxiv.org/pdf/2506.01413v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"incentivizing-reasoning-for-advanced","repo_url":"https://github.com/yuleiqin/raif","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"instruction-following","task_name":"Instruction Following"},{"task_slug":"reinforcement-learning-1","task_name":"Reinforcement Learning (RL)"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2506.01413","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2506.01413"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/yuleiqin/raif","reach":null}],"summary":{"ran_draft_wrong":3,"ran_violates":1,"unverified":2},"by_repo_kind":{"official":{"samples":6,"ran":4,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":6,"samples":[{"code_sha256_prefix":"6ef992e42dcc5d8c","entry":"extract_solution","repo":"yuleiqin/raif","repo_kind":"official","path":"openrlhf/trainer/ppo_utils/complex_scoring.py","file_url":"https://github.com/yuleiqin/raif/blob/HEAD/openrlhf/trainer/ppo_utils/complex_scoring.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"6ef992e42dcc5d8c"}},{"code_sha256_prefix":"92a9428b0e1fc30d","entry":"extract_system_user_content_from_model_input","repo":"yuleiqin/raif","repo_kind":"official","path":"openrlhf/trainer/ppo_utils/complex_scoring.py","file_url":"https://github.com/yuleiqin/raif/blob/HEAD/openrlhf/trainer/ppo_utils/complex_scoring.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"92a9428b0e1fc30d"}},{"code_sha256_prefix":"290b10d8aaab7968","entry":"shrink_system_prompt","repo":"yuleiqin/raif","repo_kind":"official","path":"openrlhf/trainer/ppo_utils/complex_scoring.py","file_url":"https://github.com/yuleiqin/raif/blob/HEAD/openrlhf/trainer/ppo_utils/complex_scoring.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"290b10d8aaab7968"}},{"code_sha256_prefix":"1ea5a95cd4fca855","entry":"validate_response_structure","repo":"yuleiqin/raif","repo_kind":"official","path":"openrlhf/trainer/ppo_utils/complex_scoring.py","file_url":"https://github.com/yuleiqin/raif/blob/HEAD/openrlhf/trainer/ppo_utils/complex_scoring.py","link_basis":"first_harvest_node","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"1ea5a95cd4fca855"}},{"code_sha256_prefix":"09f5e0b605bda802","entry":"prepare_complex_judgement","repo":"yuleiqin/raif","repo_kind":"official","path":"openrlhf/trainer/ppo_utils/complex_scoring.py","file_url":"https://github.com/yuleiqin/raif/blob/HEAD/openrlhf/trainer/ppo_utils/complex_scoring.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"09f5e0b605bda802"}},{"code_sha256_prefix":"8b40f17008e6bdda","entry":"wrapping_prompts","repo":"yuleiqin/raif","repo_kind":"official","path":"openrlhf/trainer/ppo_utils/complex_scoring.py","file_url":"https://github.com/yuleiqin/raif/blob/HEAD/openrlhf/trainer/ppo_utils/complex_scoring.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"8b40f17008e6bdda"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}