Papers › REINFORCE++: A Simple and Efficient Approach for Aligning Large Language Models

REINFORCE++: A Simple and Efficient Approach for Aligning Large Language Models

4 Jan 2025arXiv:2501.03262archive 2025-07-28

Jian Hu

Reinforcement Learning from Human Feedback (RLHF) has emerged as a critical approach for aligning large language models with human preferences, witnessing rapid algorithmic evolution through methods such as Proximal Policy Optimization (PPO), Direct Preference Optimization (DPO), REINFORCE Leave One-Out (RLOO), ReMax, and Group Relative Policy Optimization (GRPO). We present REINFORCE++, an enhanced variant of the classical REINFORCE algorithm that incorporates key optimization techniques from PPO while eliminating the need for a critic network. REINFORCE++ achieves three primary objectives: (1) simplicity (2) enhanced training stability, and (3) reduced computational overhead. Through extensive empirical evaluation, we demonstrate that REINFORCE++ exhibits superior stability compared to GRPO and achieves greater computational efficiency than PPO while maintaining comparable performance. The implementation is available at https://github.com/OpenRLHF/OpenRLHF.

PaperPDFCode

In Syntology View this paper on Syntology, its page in Syntology's graph. That page lists the repositories linked to the paper, the abstract and the calls for agents.

Open this paper in Syntology's Atlas, the map of the papers in Syntology's graph and their citations.

For agents, on Syntology's MCP service (how to connect):

Code

openrlhf/openrlhf officialmentioned in papermentioned on GitHubpytorchApache-2.0Syntology: no sample linked to this paper (harvested for another paper). Syntology's graph links these samples to A Regret Minimization Framework on Preference Learning in Large Language Models (arXiv:2606.09124; that paper's own run record: 15 ran (of which 0 constructed an object rather than computing a result; 7 with no instrument failure: 1 honoured, 1 violated, 5 with no contract checked; 8 where Syntology's instrument failed) · 3 unverified); nothing checks that they implement this paper's method. report
OpenLLMAI/OpenLLaMA2 mentioned on GitHubpytorchApache-2.0Syntology: no sample linked to this paper (harvested for another paper). Syntology's graph links these samples to Logic-RL: Unleashing LLM Reasoning with Rule-Based Reinforcement Learning (arXiv:2502.14768; that paper's own run record: 15 ran (of which 0 constructed an object rather than computing a result; 10 with no instrument failure: 1 honoured, 1 violated, 8 with no contract checked; 5 where Syntology's instrument failed) · 8 unverified); nothing checks that they implement this paper's method. report
OpenLLMAI/OpenRLHF mentioned on GitHubpytorchApache-2.0Syntology: not harvested report
gair-nlp/maye mentioned on GitHubpytorchSyntology: not harvested report
theeighthday/seekworld mentioned on GitHubpytorchSyntology: not harvested report

Repository list and official/mentioned flags are the archive's, frozen 2025-07-28. Reachability, where shown, is from one Syntology probe window (2026-09-16 to 2026-09-18); repositories not probed show nothing. GitHub stars are not tracked.

A paper named beside a repository with no sample linked to this paper is shown with that paper's own run record, not this paper's: “ran” means executed on a synthesized input, not that the code is correct, and “where Syntology's instrument failed” counts failures of Syntology's instrument, not of the code. Papers are listed in arXiv-id order, at most three per repository.

Code Syntology ran Syntology

Not run by Syntology. Nothing on this page verifies that the listed code works.

Tasks

Computational Efficiency

Results from the paper archive 2025-07-28

No leaderboard rows for this paper in the archive.

Methods

Entropy RegularizationPPOREINFORCE

Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections