{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/cream-consistency-regularized-self-rewarding","title":"CREAM: Consistency Regularized Self-Rewarding Language Models","arxiv_id":"2410.12735","date":"2024-10-16","proceeding":null,"authors":["Zhaoyang Wang","Weilei He","Zhiyuan Liang","Xuchao Zhang","Chetan Bansal","Ying WEI","Weitong Zhang","Huaxiu Yao"],"abstract":"Recent self-rewarding large language models (LLM) have successfully applied LLM-as-a-Judge to iteratively improve the alignment performance without the need of human annotations for preference data. These methods commonly utilize the same LLM to act as both the policy model (which generates responses) and the reward model (which scores and ranks those responses). The ranked responses are then used as preference pairs to train the LLM via direct alignment technologies (e.g. DPO). However, it is noteworthy that throughout this process, there is no guarantee of accuracy in the rewarding and ranking, which is critical for ensuring accurate rewards and high-quality preference data. Empirical results from relatively small LLMs (e.g., 7B parameters) also indicate that improvements from self-rewarding may diminish after several iterations in certain situations, which we hypothesize is due to accumulated bias in the reward system. This bias can lead to unreliable preference data for training the LLM. To address this issue, we first formulate and analyze the generalized iterative preference fine-tuning framework for self-rewarding language model. We then introduce the regularization to this generalized framework to mitigate the overconfident preference labeling in the self-rewarding process. Based on this theoretical insight, we propose a Consistency Regularized sElf-rewarding lAnguage Model (CREAM) that leverages the rewarding consistency across different iterations to regularize the self-rewarding training, helping the model to learn from more reliable preference data. With this explicit regularization, our empirical results demonstrate the superiority of CREAM in improving both reward consistency and alignment performance. The code is publicly available at https://github.com/Raibows/CREAM.","url_abs":"https://arxiv.org/abs/2410.12735v2","url_pdf":"https://arxiv.org/pdf/2410.12735v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"cream-consistency-regularized-self-rewarding","repo_url":"https://github.com/raibows/cream","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":"language-modeling","task_name":"Language Modeling"},{"task_slug":"language-modelling","task_name":"Language Modelling"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":null,"atlas_url":"https://app.syntology.ai/?focus=2410.12735","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2410.12735"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/raibows/cream","reach":{"status":"ok"}}],"summary":{"ran":10,"unverified":2},"by_repo_kind":{"official":{"samples":12,"ran":10,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":12,"samples":[{"code_sha256_prefix":"cd7a000c76425b3f","entry":"convert_data_to_sft","repo":"raibows/cream","repo_kind":"official","path":"data.py","file_url":"https://github.com/raibows/cream/blob/HEAD/data.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"cd7a000c76425b3f"}},{"code_sha256_prefix":"b86887299c747278","entry":"dist_broadcast_objects","repo":"raibows/cream","repo_kind":"official","path":"utils.py","file_url":"https://github.com/raibows/cream/blob/HEAD/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"b86887299c747278"}},{"code_sha256_prefix":"ed2cdc95a439ac76","entry":"dpo_forward_policy_logp","repo":"raibows/cream","repo_kind":"official","path":"sft_dpo_trainer.py","file_url":"https://github.com/raibows/cream/blob/HEAD/sft_dpo_trainer.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"ed2cdc95a439ac76"}},{"code_sha256_prefix":"eda0e2cf2fdceafc","entry":"dpo_forward_ref_logp","repo":"raibows/cream","repo_kind":"official","path":"sft_dpo_trainer.py","file_url":"https://github.com/raibows/cream/blob/HEAD/sft_dpo_trainer.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"eda0e2cf2fdceafc"}},{"code_sha256_prefix":"7eaf6f1c387075b1","entry":"extract_qa_final_answer","repo":"raibows/cream","repo_kind":"official","path":"evaluate.py","file_url":"https://github.com/raibows/cream/blob/HEAD/evaluate.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"7eaf6f1c387075b1"}},{"code_sha256_prefix":"fc49b73a5d91cb17","entry":"get_self_rewarding_text_prompt","repo":"raibows/cream","repo_kind":"official","path":"prompts.py","file_url":"https://github.com/raibows/cream/blob/HEAD/prompts.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"fc49b73a5d91cb17"}},{"code_sha256_prefix":"a989e32e3398c762","entry":"gsm8k_extract_final_answer","repo":"raibows/cream","repo_kind":"official","path":"evaluate.py","file_url":"https://github.com/raibows/cream/blob/HEAD/evaluate.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"a989e32e3398c762"}},{"code_sha256_prefix":"6c89dab673389760","entry":"tools_get_model_name","repo":"raibows/cream","repo_kind":"official","path":"tools.py","file_url":"https://github.com/raibows/cream/blob/HEAD/tools.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"6c89dab673389760"}},{"code_sha256_prefix":"f86c1e46974e50f2","entry":"tools_is_device_cpu","repo":"raibows/cream","repo_kind":"official","path":"tools.py","file_url":"https://github.com/raibows/cream/blob/HEAD/tools.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"f86c1e46974e50f2"}},{"code_sha256_prefix":"d9de59ec19277b02","entry":"tools_json_load","repo":"raibows/cream","repo_kind":"official","path":"tools.py","file_url":"https://github.com/raibows/cream/blob/HEAD/tools.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"d9de59ec19277b02"}},{"code_sha256_prefix":"70d2c51f19cf7e48","entry":"dist_sync_objects","repo":"raibows/cream","repo_kind":"official","path":"utils.py","file_url":"https://github.com/raibows/cream/blob/HEAD/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"70d2c51f19cf7e48"}},{"code_sha256_prefix":"9bbe9b4a08066682","entry":"get_hf_training_args","repo":"raibows/cream","repo_kind":"official","path":"config.py","file_url":"https://github.com/raibows/cream/blob/HEAD/config.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"9bbe9b4a08066682"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}