{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/coevolving-with-the-other-you-fine-tuning-llm","title":"Coevolving with the Other You: Fine-Tuning LLM with Sequential Cooperative Multi-Agent Reinforcement Learning","arxiv_id":"2410.06101","date":"2024-10-08","proceeding":null,"authors":["Hao Ma","Tianyi Hu","Zhiqiang Pu","Boyin Liu","Xiaolin Ai","Yanyan Liang","Min Chen"],"abstract":"Reinforcement learning (RL) has emerged as a pivotal technique for fine-tuning large language models (LLMs) on specific tasks. However, prevailing RL fine-tuning methods predominantly rely on PPO and its variants. Though these algorithms are effective in general RL settings, they often exhibit suboptimal performance and vulnerability to distribution collapse when applied to the fine-tuning of LLMs. In this paper, we propose CORY, extending the RL fine-tuning of LLMs to a sequential cooperative multi-agent reinforcement learning framework, to leverage the inherent coevolution and emergent capabilities of multi-agent systems. In CORY, the LLM to be fine-tuned is initially duplicated into two autonomous agents: a pioneer and an observer. The pioneer generates responses based on queries, while the observer generates responses using both the queries and the pioneer's responses. The two agents are trained together. During training, the agents exchange roles periodically, fostering cooperation and coevolution between them. Experiments evaluate CORY's performance by fine-tuning GPT-2 and Llama-2 under subjective and objective reward functions on the IMDB Review and GSM8K datasets, respectively. Results show that CORY outperforms PPO in terms of policy optimality, resistance to distribution collapse, and training robustness, thereby underscoring its potential as a superior methodology for refining LLMs in real-world applications.","url_abs":"https://arxiv.org/abs/2410.06101v2","url_pdf":"https://arxiv.org/pdf/2410.06101v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"coevolving-with-the-other-you-fine-tuning-llm","repo_url":"https://github.com/Harry67Hu/CORY","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"gsm8k","task_name":"GSM8K"},{"task_slug":"multi-agent-reinforcement-learning","task_name":"Multi-agent Reinforcement Learning"},{"task_slug":"reinforcement-learning","task_name":"Reinforcement Learning"},{"task_slug":"reinforcement-learning-1","task_name":"Reinforcement Learning (RL)"},{"task_slug":"reinforcement-learning-2","task_name":"reinforcement-learning"}],"methods":[{"method_slug":"adam","method_name":"Adam"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"attention-dropout","method_name":"Attention Dropout"},{"method_slug":"bpe","method_name":"BPE"},{"method_slug":"cosine-annealing","method_name":"Cosine Annealing"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"discriminative-fine-tuning","method_name":"Discriminative Fine-Tuning"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"entropy-regularization","method_name":"Entropy Regularization"},{"method_slug":"gpt-2","method_name":"GPT-2"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"linear-warmup-with-cosine-annealing","method_name":"Linear Warmup With Cosine Annealing"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"ppo","method_name":"PPO"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"weight-decay","method_name":"Weight Decay"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2410.06101","atlas_url":"https://app.syntology.ai/?focus=2410.06101","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2410.06101"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/Harry67Hu/CORY","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran_draft_wrong":2,"ran":5,"unverified":4},"by_repo_kind":{"official":{"samples":11,"ran":7,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"bda36ad0c44dbd74","entry":"collator","repo":"Harry67Hu/CORY","repo_kind":"official","path":"imdb_train/cory.py","file_url":"https://github.com/Harry67Hu/CORY/blob/HEAD/imdb_train/cory.py","link_basis":"plan_row","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"bda36ad0c44dbd74"}},{"code_sha256_prefix":"c32b636f13ed9ee5","entry":"convert_to_scalar","repo":"Harry67Hu/CORY","repo_kind":"official","path":"trl/core.py","file_url":"https://github.com/Harry67Hu/CORY/blob/HEAD/trl/core.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c32b636f13ed9ee5"}},{"code_sha256_prefix":"5013cfeaf23a03ad","entry":"create_demo_text","repo":"Harry67Hu/CORY","repo_kind":"official","path":"gsm8k_utils/gsm8k_utils.py","file_url":"https://github.com/Harry67Hu/CORY/blob/HEAD/gsm8k_utils/gsm8k_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"5013cfeaf23a03ad"}},{"code_sha256_prefix":"96fe095fcc6b821a","entry":"extract_answer_from_output","repo":"Harry67Hu/CORY","repo_kind":"official","path":"gsm8k_utils/gsm8k_utils.py","file_url":"https://github.com/Harry67Hu/CORY/blob/HEAD/gsm8k_utils/gsm8k_utils.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"96fe095fcc6b821a"}},{"code_sha256_prefix":"e198da7bdc5637e3","entry":"is_correct","repo":"Harry67Hu/CORY","repo_kind":"official","path":"gsm8k_utils/gsm8k_utils.py","file_url":"https://github.com/Harry67Hu/CORY/blob/HEAD/gsm8k_utils/gsm8k_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e198da7bdc5637e3"}},{"code_sha256_prefix":"3fe7529a76b17e9c","entry":"respond_to_batch","repo":"Harry67Hu/CORY","repo_kind":"official","path":"trl/models/model.py","file_url":"https://github.com/Harry67Hu/CORY/blob/HEAD/trl/models/model.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"3fe7529a76b17e9c"}},{"code_sha256_prefix":"5bcf8e086423cf6b","entry":"top_k_top_p_filtering","repo":"Harry67Hu/CORY","repo_kind":"official","path":"trl/core.py","file_url":"https://github.com/Harry67Hu/CORY/blob/HEAD/trl/core.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"5bcf8e086423cf6b"}},{"code_sha256_prefix":"0efa9af73116efc0","entry":"build_dataset","repo":"Harry67Hu/CORY","repo_kind":"official","path":"gsm8k_utils/gsm8k_eval.py","file_url":"https://github.com/Harry67Hu/CORY/blob/HEAD/gsm8k_utils/gsm8k_eval.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"0efa9af73116efc0"}},{"code_sha256_prefix":"a51ae45a1f6c0b5b","entry":"flatten_dict","repo":"Harry67Hu/CORY","repo_kind":"official","path":"trl/core.py","file_url":"https://github.com/Harry67Hu/CORY/blob/HEAD/trl/core.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a51ae45a1f6c0b5b"}},{"code_sha256_prefix":"f4334934ee795269","entry":"is_transformers_greater_than","repo":"Harry67Hu/CORY","repo_kind":"official","path":"trl/import_utils.py","file_url":"https://github.com/Harry67Hu/CORY/blob/HEAD/trl/import_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f4334934ee795269"}},{"code_sha256_prefix":"59046fe10dd792f4","entry":"scheduler_step","repo":"Harry67Hu/CORY","repo_kind":"official","path":"trl/models/modeling_sd_base.py","file_url":"https://github.com/Harry67Hu/CORY/blob/HEAD/trl/models/modeling_sd_base.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"59046fe10dd792f4"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}