{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/unfamiliar-finetuning-examples-control-how","title":"Unfamiliar Finetuning Examples Control How Language Models Hallucinate","arxiv_id":"2403.05612","date":"2024-03-08","proceeding":null,"authors":["Katie Kang","Eric Wallace","Claire Tomlin","Aviral Kumar","Sergey Levine"],"abstract":"Large language models are known to hallucinate when faced with unfamiliar queries, but the underlying mechanism that govern how models hallucinate are not yet fully understood. In this work, we find that unfamiliar examples in the models' finetuning data -- those that introduce concepts beyond the base model's scope of knowledge -- are crucial in shaping these errors. In particular, we find that an LLM's hallucinated predictions tend to mirror the responses associated with its unfamiliar finetuning examples. This suggests that by modifying how unfamiliar finetuning examples are supervised, we can influence a model's responses to unfamiliar queries (e.g., say ``I don't know''). We empirically validate this observation in a series of controlled experiments involving SFT, RL, and reward model finetuning on TriviaQA and MMLU. Our work further investigates RL finetuning strategies for improving the factuality of long-form model generations. We find that, while hallucinations from the reward model can significantly undermine the effectiveness of RL factuality finetuning, strategically controlling how reward models hallucinate can minimize these negative effects. Leveraging our previous observations on controlling hallucinations, we propose an approach for learning more reliable reward models, and show that they improve the efficacy of RL factuality finetuning in long-form biography and book/movie plot generation tasks.","url_abs":"https://arxiv.org/abs/2403.05612v2","url_pdf":"https://arxiv.org/pdf/2403.05612v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"unfamiliar-finetuning-examples-control-how","repo_url":"https://github.com/katiekang1998/llm_hallucinations","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":"mmlu","task_name":"MMLU"},{"task_slug":"multiple-choice","task_name":"Multiple-choice"},{"task_slug":"triviaqa","task_name":"TriviaQA"}],"methods":[{"method_slug":"base","method_name":"BASE"},{"method_slug":"sft","method_name":"SFT"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2403.05612","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2403.05612"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/katiekang1998/llm_hallucinations","reach":{"status":"ok"}}],"summary":{"ran":5,"unverified":2},"by_repo_kind":{"official":{"samples":7,"ran":5,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":7,"samples":[{"code_sha256_prefix":"d552ae8566ea9d61","entry":"get_param_space","repo":"katiekang1998/llm_hallucinations","repo_kind":"official","path":"trlx/sweep.py","file_url":"https://github.com/katiekang1998/llm_hallucinations/blob/HEAD/trlx/sweep.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"d552ae8566ea9d61"}},{"code_sha256_prefix":"8fae271bba35cd50","entry":"get_scheduler","repo":"katiekang1998/llm_hallucinations","repo_kind":"official","path":"trlx/sweep.py","file_url":"https://github.com/katiekang1998/llm_hallucinations/blob/HEAD/trlx/sweep.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"8fae271bba35cd50"}},{"code_sha256_prefix":"2ab02dee00149590","entry":"make_head","repo":"katiekang1998/llm_hallucinations","repo_kind":"official","path":"trlx/utils/modeling.py","file_url":"https://github.com/katiekang1998/llm_hallucinations/blob/HEAD/trlx/utils/modeling.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"2ab02dee00149590"}},{"code_sha256_prefix":"ace162af010af5d8","entry":"rhasattr","repo":"katiekang1998/llm_hallucinations","repo_kind":"official","path":"trlx/utils/modeling.py","file_url":"https://github.com/katiekang1998/llm_hallucinations/blob/HEAD/trlx/utils/modeling.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"ace162af010af5d8"}},{"code_sha256_prefix":"3bfb7deb629cb756","entry":"topk_mask","repo":"katiekang1998/llm_hallucinations","repo_kind":"official","path":"trlx/models/modeling_ilql.py","file_url":"https://github.com/katiekang1998/llm_hallucinations/blob/HEAD/trlx/models/modeling_ilql.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"3bfb7deb629cb756"}},{"code_sha256_prefix":"690d90daad365060","entry":"get_search_alg","repo":"katiekang1998/llm_hallucinations","repo_kind":"official","path":"trlx/sweep.py","file_url":"https://github.com/katiekang1998/llm_hallucinations/blob/HEAD/trlx/sweep.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"690d90daad365060"}},{"code_sha256_prefix":"fc2499cfa7162602","entry":"rgetattr","repo":"katiekang1998/llm_hallucinations","repo_kind":"official","path":"trlx/utils/modeling.py","file_url":"https://github.com/katiekang1998/llm_hallucinations/blob/HEAD/trlx/utils/modeling.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"fc2499cfa7162602"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}