{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/trustworthy-alignment-of-retrieval-augmented","title":"Trustworthy Alignment of Retrieval-Augmented Large Language Models via Reinforcement Learning","arxiv_id":"2410.16843","date":"2024-10-22","proceeding":null,"authors":["Zongmeng Zhang","Yufeng Shi","Jinhua Zhu","Wengang Zhou","Xiang Qi","Peng Zhang","Houqiang Li"],"abstract":"Trustworthiness is an essential prerequisite for the real-world application of large language models. In this paper, we focus on the trustworthiness of language models with respect to retrieval augmentation. Despite being supported with external evidence, retrieval-augmented generation still suffers from hallucinations, one primary cause of which is the conflict between contextual and parametric knowledge. We deem that retrieval-augmented language models have the inherent capabilities of supplying response according to both contextual and parametric knowledge. Inspired by aligning language models with human preference, we take the first step towards aligning retrieval-augmented language models to a status where it responds relying merely on the external evidence and disregards the interference of parametric knowledge. Specifically, we propose a reinforcement learning based algorithm Trustworthy-Alignment, theoretically and experimentally demonstrating large language models' capability of reaching a trustworthy status without explicit supervision on how to respond. Our work highlights the potential of large language models on exploring its intrinsic abilities by its own and expands the application scenarios of alignment from fulfilling human preference to creating trustworthy agents.","url_abs":"https://arxiv.org/abs/2410.16843v1","url_pdf":"https://arxiv.org/pdf/2410.16843v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"trustworthy-alignment-of-retrieval-augmented","repo_url":"https://github.com/zmzhang2000/trustworthy-alignment","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":"retrieval","task_name":"Retrieval"},{"task_slug":"retrieval-augmented-generation","task_name":"Retrieval-augmented Generation"}],"methods":[{"method_slug":"focus","method_name":"Focus"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2410.16843","atlas_url":"https://app.syntology.ai/?focus=2410.16843","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2410.16843"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/zmzhang2000/trustworthy-alignment","reach":{"status":"ok"}}],"summary":{"ran_fixture":1,"unverified":12},"by_repo_kind":{"official":{"samples":13,"ran":1,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":13,"samples":[{"code_sha256_prefix":"2f0a690c70d11c23","entry":"gather_log_probs","repo":"zmzhang2000/trustworthy-alignment","repo_kind":"official","path":"dschat/rlhf/ppo_trainer.py","file_url":"https://github.com/zmzhang2000/trustworthy-alignment/blob/HEAD/dschat/rlhf/ppo_trainer.py","link_basis":"plan_row","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"2f0a690c70d11c23"}},{"code_sha256_prefix":"0aadf110b32fdf12","entry":"best_subspan_em","repo":"zmzhang2000/trustworthy-alignment","repo_kind":"official","path":"dschat/utils/metrics.py","file_url":"https://github.com/zmzhang2000/trustworthy-alignment/blob/HEAD/dschat/utils/metrics.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"0aadf110b32fdf12"}},{"code_sha256_prefix":"069f56cb00324e10","entry":"calculate_flops","repo":"zmzhang2000/trustworthy-alignment","repo_kind":"official","path":"dschat/utils/perf.py","file_url":"https://github.com/zmzhang2000/trustworthy-alignment/blob/HEAD/dschat/utils/perf.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"069f56cb00324e10"}},{"code_sha256_prefix":"b6934717c02ba140","entry":"causal_lm_model_to_fp32_loss","repo":"zmzhang2000/trustworthy-alignment","repo_kind":"official","path":"dschat/utils/model/model_utils.py","file_url":"https://github.com/zmzhang2000/trustworthy-alignment/blob/HEAD/dschat/utils/model/model_utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"b6934717c02ba140"}},{"code_sha256_prefix":"4d2823b65af0c2df","entry":"create_hf_model","repo":"zmzhang2000/trustworthy-alignment","repo_kind":"official","path":"dschat/utils/model/model_utils.py","file_url":"https://github.com/zmzhang2000/trustworthy-alignment/blob/HEAD/dschat/utils/model/model_utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"4d2823b65af0c2df"}},{"code_sha256_prefix":"eff7424deef19565","entry":"generate_qa_attributed_prompt","repo":"zmzhang2000/trustworthy-alignment","repo_kind":"official","path":"dschat/utils/prompts.py","file_url":"https://github.com/zmzhang2000/trustworthy-alignment/blob/HEAD/dschat/utils/prompts.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"eff7424deef19565"}},{"code_sha256_prefix":"9a6cc185b5239af7","entry":"generate_qa_base_prompt","repo":"zmzhang2000/trustworthy-alignment","repo_kind":"official","path":"dschat/utils/prompts.py","file_url":"https://github.com/zmzhang2000/trustworthy-alignment/blob/HEAD/dschat/utils/prompts.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"9a6cc185b5239af7"}},{"code_sha256_prefix":"062abf04b8fabe82","entry":"generate_qa_opinion_prompt","repo":"zmzhang2000/trustworthy-alignment","repo_kind":"official","path":"dschat/utils/prompts.py","file_url":"https://github.com/zmzhang2000/trustworthy-alignment/blob/HEAD/dschat/utils/prompts.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"062abf04b8fabe82"}},{"code_sha256_prefix":"792e87a1badc5642","entry":"get_eval_ds_config","repo":"zmzhang2000/trustworthy-alignment","repo_kind":"official","path":"dschat/utils/ds_utils.py","file_url":"https://github.com/zmzhang2000/trustworthy-alignment/blob/HEAD/dschat/utils/ds_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"792e87a1badc5642"}},{"code_sha256_prefix":"df20c2f4083d17af","entry":"get_hf_configs","repo":"zmzhang2000/trustworthy-alignment","repo_kind":"official","path":"dschat/utils/perf.py","file_url":"https://github.com/zmzhang2000/trustworthy-alignment/blob/HEAD/dschat/utils/perf.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"df20c2f4083d17af"}},{"code_sha256_prefix":"429d42a3bf925da5","entry":"log_init","repo":"zmzhang2000/trustworthy-alignment","repo_kind":"official","path":"dschat/rlhf/rlhf_engine.py","file_url":"https://github.com/zmzhang2000/trustworthy-alignment/blob/HEAD/dschat/rlhf/rlhf_engine.py","link_basis":"plan_row","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"429d42a3bf925da5"}},{"code_sha256_prefix":"cf3df3743796f762","entry":"multi_choice_hit","repo":"zmzhang2000/trustworthy-alignment","repo_kind":"official","path":"dschat/utils/metrics.py","file_url":"https://github.com/zmzhang2000/trustworthy-alignment/blob/HEAD/dschat/utils/metrics.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"cf3df3743796f762"}},{"code_sha256_prefix":"9706a9a0b961096b","entry":"normalize_answer","repo":"zmzhang2000/trustworthy-alignment","repo_kind":"official","path":"dschat/utils/metrics.py","file_url":"https://github.com/zmzhang2000/trustworthy-alignment/blob/HEAD/dschat/utils/metrics.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"9706a9a0b961096b"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}