{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/lire-listwise-reward-enhancement-for","title":"LIRE: listwise reward enhancement for preference alignment","arxiv_id":"2405.13516","date":"2024-05-22","proceeding":null,"authors":["Mingye Zhu","Yi Liu","Lei Zhang","Junbo Guo","Zhendong Mao"],"abstract":"Recently, tremendous strides have been made to align the generation of Large Language Models (LLMs) with human values to mitigate toxic or unhelpful content. Leveraging Reinforcement Learning from Human Feedback (RLHF) proves effective and is widely adopted by researchers. However, implementing RLHF is complex, and its sensitivity to hyperparameters renders achieving stable performance and scalability challenging. Furthermore, prevailing approaches to preference alignment primarily concentrate on pairwise comparisons, with limited exploration into multi-response scenarios, thereby overlooking the potential richness within the candidate pool. For the above reasons, we propose a new approach: Listwise Reward Enhancement for Preference Alignment (LIRE), a gradient-based reward optimization approach that incorporates the offline rewards of multiple responses into a streamlined listwise framework, thus eliminating the need for online sampling during training. LIRE is straightforward to implement, requiring minimal parameter tuning, and seamlessly aligns with the pairwise paradigm while naturally extending to multi-response scenarios. Moreover, we introduce a self-enhancement algorithm aimed at iteratively refining the reward during training. Our experiments demonstrate that LIRE consistently outperforms existing methods across several benchmarks on dialogue and summarization tasks, with good transferability to out-of-distribution data, assessed using proxy reward models and human annotators.","url_abs":"https://arxiv.org/abs/2405.13516v2","url_pdf":"https://arxiv.org/pdf/2405.13516v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"lire-listwise-reward-enhancement-for","repo_url":"https://github.com/stevie1023/LIRE","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[],"methods":[{"method_slug":"align","method_name":"ALIGN"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2405.13516","atlas_url":"https://app.syntology.ai/?focus=2405.13516","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2405.13516"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/stevie1023/LIRE","reach":{"status":"ok"}}],"summary":{"ran_draft_wrong":3,"ran":2,"ran_honours":1},"by_repo_kind":{"official":{"samples":6,"ran":6,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":6,"samples":[{"code_sha256_prefix":"d07d04439cd1d44f","entry":"jload","repo":"stevie1023/LIRE","repo_kind":"official","path":"train_alpaca_prompt.py","file_url":"https://github.com/stevie1023/LIRE/blob/HEAD/train_alpaca_prompt.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"d07d04439cd1d44f"}},{"code_sha256_prefix":"ce6dbeeae3d77963","entry":"make_supervised_data_module","repo":"stevie1023/LIRE","repo_kind":"official","path":"train_alpaca_prompt.py","file_url":"https://github.com/stevie1023/LIRE/blob/HEAD/train_alpaca_prompt.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"ce6dbeeae3d77963"}},{"code_sha256_prefix":"741b411ba8cc7fcb","entry":"padding","repo":"stevie1023/LIRE","repo_kind":"official","path":"data_generation/response_gen.py","file_url":"https://github.com/stevie1023/LIRE/blob/HEAD/data_generation/response_gen.py","link_basis":"plan_row","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"741b411ba8cc7fcb"}},{"code_sha256_prefix":"5e7416d3c1bc76ee","entry":"preprocess","repo":"stevie1023/LIRE","repo_kind":"official","path":"data_generation/response_gen.py","file_url":"https://github.com/stevie1023/LIRE/blob/HEAD/data_generation/response_gen.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"5e7416d3c1bc76ee"}},{"code_sha256_prefix":"3a39317f52858530","entry":"sequence_gather","repo":"stevie1023/LIRE","repo_kind":"official","path":"data_generation/response_gen.py","file_url":"https://github.com/stevie1023/LIRE/blob/HEAD/data_generation/response_gen.py","link_basis":"plan_row","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"3a39317f52858530"}},{"code_sha256_prefix":"d31569d4019ad09b","entry":"stop_response","repo":"stevie1023/LIRE","repo_kind":"official","path":"train_alpaca_prompt.py","file_url":"https://github.com/stevie1023/LIRE/blob/HEAD/train_alpaca_prompt.py","link_basis":"plan_row","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"d31569d4019ad09b"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}