{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/echo-chamber-rl-post-training-amplifies","title":"Echo Chamber: RL Post-training Amplifies Behaviors Learned in Pretraining","arxiv_id":"2504.07912","date":"2025-04-10","proceeding":null,"authors":["Rosie Zhao","Alexandru Meterez","Sham Kakade","Cengiz Pehlevan","Samy Jelassi","Eran Malach"],"abstract":"Reinforcement learning (RL)-based fine-tuning has become a crucial step in post-training language models for advanced mathematical reasoning and coding. Following the success of frontier reasoning models, recent work has demonstrated that RL fine-tuning consistently improves performance, even in smaller-scale models; however, the underlying mechanisms driving these improvements are not well-understood. Understanding the effects of RL fine-tuning requires disentangling its interaction with pretraining data composition, hyperparameters, and model scale, but such problems are exacerbated by the lack of transparency regarding the training data used in many existing models. In this work, we present a systematic end-to-end study of RL fine-tuning for mathematical reasoning by training models entirely from scratch on different mixtures of fully open datasets. We investigate the effects of various RL fine-tuning algorithms (PPO, GRPO, and Expert Iteration) across models of different scales. Our study reveals that RL algorithms consistently converge towards a dominant output distribution, amplifying patterns in the pretraining data. We also find that models of different scales trained on the same data mixture will converge to distinct output distributions, suggesting that there are scale-dependent biases in model generalization. Moreover, we find that RL post-training on simpler questions can lead to performance gains on harder ones, indicating that certain reasoning capabilities generalize across tasks. Our findings show that small-scale proxies in controlled settings can elicit interesting insights regarding the role of RL in shaping language model behavior.","url_abs":"https://arxiv.org/abs/2504.07912v1","url_pdf":"https://arxiv.org/pdf/2504.07912v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"echo-chamber-rl-post-training-amplifies","repo_url":"https://github.com/rosieyzh/openrlhf-pretrain","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"mathematical-reasoning","task_name":"Mathematical Reasoning"},{"task_slug":"reinforcement-learning-1","task_name":"Reinforcement Learning (RL)"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2504.07912","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2504.07912"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/rosieyzh/openrlhf-pretrain","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran":1,"unverified":6},"by_repo_kind":{"official":{"samples":7,"ran":1,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"75afa2affd2a8e36","entry":"log_probs_from_logits","repo":"rosieyzh/openrlhf-pretrain","repo_kind":"official","path":"openrlhf/models/utils.py","file_url":"https://github.com/rosieyzh/openrlhf-pretrain/blob/HEAD/openrlhf/models/utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"75afa2affd2a8e36"}},{"code_sha256_prefix":"95084b4b49902e3d","entry":"compute_approx_kl","repo":"rosieyzh/openrlhf-pretrain","repo_kind":"official","path":"openrlhf/models/utils.py","file_url":"https://github.com/rosieyzh/openrlhf-pretrain/blob/HEAD/openrlhf/models/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"95084b4b49902e3d"}},{"code_sha256_prefix":"0d27f3b8c425d79f","entry":"compute_reward","repo":"rosieyzh/openrlhf-pretrain","repo_kind":"official","path":"openrlhf/models/utils.py","file_url":"https://github.com/rosieyzh/openrlhf-pretrain/blob/HEAD/openrlhf/models/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"0d27f3b8c425d79f"}},{"code_sha256_prefix":"d66f259aaf6cd9c7","entry":"convert_ring_attn_params","repo":"rosieyzh/openrlhf-pretrain","repo_kind":"official","path":"openrlhf/models/ring_attn_utils.py","file_url":"https://github.com/rosieyzh/openrlhf-pretrain/blob/HEAD/openrlhf/models/ring_attn_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"d66f259aaf6cd9c7"}},{"code_sha256_prefix":"b26627fa753393a8","entry":"pad_sequences","repo":"rosieyzh/openrlhf-pretrain","repo_kind":"official","path":"openrlhf/models/ring_attn_utils.py","file_url":"https://github.com/rosieyzh/openrlhf-pretrain/blob/HEAD/openrlhf/models/ring_attn_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"b26627fa753393a8"}},{"code_sha256_prefix":"e93117d99aa6b210","entry":"reset_ring_attn_position_ids","repo":"rosieyzh/openrlhf-pretrain","repo_kind":"official","path":"openrlhf/models/ring_attn_utils.py","file_url":"https://github.com/rosieyzh/openrlhf-pretrain/blob/HEAD/openrlhf/models/ring_attn_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"e93117d99aa6b210"}},{"code_sha256_prefix":"ca3514c46a1261d7","entry":"strip_sequence","repo":"rosieyzh/openrlhf-pretrain","repo_kind":"official","path":"openrlhf/cli/serve_rm.py","file_url":"https://github.com/rosieyzh/openrlhf-pretrain/blob/HEAD/openrlhf/cli/serve_rm.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"ca3514c46a1261d7"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}