{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/salmon-self-alignment-with-principle","title":"SALMON: Self-Alignment with Instructable Reward Models","arxiv_id":"2310.05910","date":"2023-10-09","proceeding":null,"authors":["Zhiqing Sun","Yikang Shen","Hongxin Zhang","Qinhong Zhou","Zhenfang Chen","David Cox","Yiming Yang","Chuang Gan"],"abstract":"Supervised Fine-Tuning (SFT) on response demonstrations combined with Reinforcement Learning from Human Feedback (RLHF) constitutes a powerful paradigm for aligning LLM-based AI agents. However, a significant limitation of such an approach is its dependency on high-quality human annotations, making its application to intricate tasks challenging due to difficulties in obtaining consistent response demonstrations and in-distribution response preferences. This paper presents a novel approach, namely SALMON, to align base language models with minimal human supervision, using only a small set of human-defined principles, yet achieving superior performance. Central to our approach is an instructable reward model. Trained on synthetic preference data, this model can generate reward scores based on arbitrary human-defined principles. By merely adjusting these principles during the RL training phase, we gain full control over the preferences with the instructable reward model, subsequently influencing the behavior of the RL-trained policy models, and reducing the reliance on the collection of online human preferences. Applying our method to the LLaMA-2-70b base language model, we developed an AI assistant named Dromedary-2. With only 6 exemplars for in-context learning and 31 human-defined principles, Dromedary-2 significantly surpasses the performance of several state-of-the-art AI systems, including LLaMA-2-Chat-70b, on various benchmark datasets. We have open-sourced the code and model weights to encourage further research into aligning LLM-based AI agents with enhanced supervision efficiency, improved controllability, and scalable oversight.","url_abs":"https://arxiv.org/abs/2310.05910v2","url_pdf":"https://arxiv.org/pdf/2310.05910v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"salmon-self-alignment-with-principle","repo_url":"https://github.com/ibm/salmon","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"GPL-3.0"}}],"tasks":[{"task_slug":"in-context-learning","task_name":"In-Context Learning"},{"task_slug":"language-modelling","task_name":"Language Modelling"}],"methods":[{"method_slug":"align","method_name":"ALIGN"},{"method_slug":"base","method_name":"BASE"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2310.05910","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2310.05910"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/ibm/salmon","reach":{"status":"ok","spdx":"GPL-3.0"}}],"summary":{"ran_draft_wrong":4,"ran_fixture":1,"ran":8},"by_repo_kind":{"official":{"samples":13,"ran":13,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":13,"samples":[{"code_sha256_prefix":"2154e44a6b59e432","entry":"batch_select","repo":"ibm/salmon","repo_kind":"official","path":"training/models/reward_model.py","file_url":"https://github.com/ibm/salmon/blob/HEAD/training/models/reward_model.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":2,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"GPL-3.0","inline_ok":false,"mcp_get_code":{"code_sha256":"2154e44a6b59e432"}},{"code_sha256_prefix":"30d7eec482ebf6b1","entry":"repeat_kv","repo":"ibm/salmon","repo_kind":"official","path":"training/models/llama_with_flash_attn.py","file_url":"https://github.com/ibm/salmon/blob/HEAD/training/models/llama_with_flash_attn.py","link_basis":"harvester_set","language":"python","status":"ran_fixture","verification_level":2,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"GPL-3.0","inline_ok":false,"mcp_get_code":{"code_sha256":"30d7eec482ebf6b1"}},{"code_sha256_prefix":"a6989412aca721c1","entry":"unpack_dict","repo":"ibm/salmon","repo_kind":"official","path":"training/models/reward_model.py","file_url":"https://github.com/ibm/salmon/blob/HEAD/training/models/reward_model.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":2,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"GPL-3.0","inline_ok":false,"mcp_get_code":{"code_sha256":"a6989412aca721c1"}},{"code_sha256_prefix":"5aa59c85970b28a7","entry":"all_gather_and_cat","repo":"ibm/salmon","repo_kind":"official","path":"training/models/distributed_utils.py","file_url":"https://github.com/ibm/salmon/blob/HEAD/training/models/distributed_utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"GPL-3.0","inline_ok":false,"mcp_get_code":{"code_sha256":"5aa59c85970b28a7"}},{"code_sha256_prefix":"e34097675d132bc1","entry":"apply_rotary_pos_emb","repo":"ibm/salmon","repo_kind":"official","path":"training/models/llama_with_flash_attn.py","file_url":"https://github.com/ibm/salmon/blob/HEAD/training/models/llama_with_flash_attn.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"GPL-3.0","inline_ok":false,"mcp_get_code":{"code_sha256":"e34097675d132bc1"}},{"code_sha256_prefix":"af2735cffc6ee536","entry":"cast_with_native_amp","repo":"ibm/salmon","repo_kind":"official","path":"training/models/rl_trainer.py","file_url":"https://github.com/ibm/salmon/blob/HEAD/training/models/rl_trainer.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"GPL-3.0","inline_ok":false,"mcp_get_code":{"code_sha256":"af2735cffc6ee536"}},{"code_sha256_prefix":"d5a396157801a324","entry":"clean_after_stop_token","repo":"ibm/salmon","repo_kind":"official","path":"training/models/ppo_trainer.py","file_url":"https://github.com/ibm/salmon/blob/HEAD/training/models/ppo_trainer.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"GPL-3.0","inline_ok":false,"mcp_get_code":{"code_sha256":"d5a396157801a324"}},{"code_sha256_prefix":"42596ac916e883a5","entry":"create_scheduler","repo":"ibm/salmon","repo_kind":"official","path":"training/models/trainer_utils.py","file_url":"https://github.com/ibm/salmon/blob/HEAD/training/models/trainer_utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"GPL-3.0","inline_ok":false,"mcp_get_code":{"code_sha256":"42596ac916e883a5"}},{"code_sha256_prefix":"89ce33341af66037","entry":"get_last_checkpoint","repo":"ibm/salmon","repo_kind":"official","path":"training/qlora_utils.py","file_url":"https://github.com/ibm/salmon/blob/HEAD/training/qlora_utils.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"GPL-3.0","inline_ok":false,"mcp_get_code":{"code_sha256":"89ce33341af66037"}},{"code_sha256_prefix":"b99eea6376d1e212","entry":"rotate_half","repo":"ibm/salmon","repo_kind":"official","path":"training/models/llama_with_flash_attn.py","file_url":"https://github.com/ibm/salmon/blob/HEAD/training/models/llama_with_flash_attn.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"GPL-3.0","inline_ok":false,"mcp_get_code":{"code_sha256":"b99eea6376d1e212"}},{"code_sha256_prefix":"4098e9c2afde4a5d","entry":"setup","repo":"ibm/salmon","repo_kind":"official","path":"training/models/distributed_utils.py","file_url":"https://github.com/ibm/salmon/blob/HEAD/training/models/distributed_utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"GPL-3.0","inline_ok":false,"mcp_get_code":{"code_sha256":"4098e9c2afde4a5d"}},{"code_sha256_prefix":"1088838ef514bf22","entry":"truncate_after_eos","repo":"ibm/salmon","repo_kind":"official","path":"training/models/rl_trainer.py","file_url":"https://github.com/ibm/salmon/blob/HEAD/training/models/rl_trainer.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"GPL-3.0","inline_ok":false,"mcp_get_code":{"code_sha256":"1088838ef514bf22"}},{"code_sha256_prefix":"9a0b92f09a2d6c76","entry":"whiten","repo":"ibm/salmon","repo_kind":"official","path":"training/models/ppo_trainer.py","file_url":"https://github.com/ibm/salmon/blob/HEAD/training/models/ppo_trainer.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"GPL-3.0","inline_ok":false,"mcp_get_code":{"code_sha256":"9a0b92f09a2d6c76"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}