{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/targeted-manipulation-and-deception-emerge","title":"On Targeted Manipulation and Deception when Optimizing LLMs for User Feedback","arxiv_id":"2411.02306","date":"2024-11-04","proceeding":null,"authors":["Marcus Williams","Micah Carroll","Adhyyan Narang","Constantin Weisser","Brendan Murphy","Anca Dragan"],"abstract":"As LLMs become more widely deployed, there is increasing interest in directly optimizing for feedback from end users (e.g. thumbs up) in addition to feedback from paid annotators. However, training to maximize human feedback creates a perverse incentive structure for the AI to resort to manipulative or deceptive tactics to obtain positive feedback from users who are vulnerable to such strategies. We study this phenomenon by training LLMs with Reinforcement Learning with simulated user feedback in environments of practical LLM usage. In our settings, we find that: 1) Extreme forms of \"feedback gaming\" such as manipulation and deception are learned reliably; 2) Even if only 2% of users are vulnerable to manipulative strategies, LLMs learn to identify and target them while behaving appropriately with other users, making such behaviors harder to detect; 3) To mitigate this issue, it may seem promising to leverage continued safety training or LLM-as-judges during training to filter problematic outputs. Instead, we found that while such approaches help in some of our settings, they backfire in others, sometimes even leading to subtler manipulative behaviors. We hope our results can serve as a case study which highlights the risks of using gameable feedback sources -- such as user feedback -- as a target for RL.","url_abs":"https://arxiv.org/abs/2411.02306v2","url_pdf":"https://arxiv.org/pdf/2411.02306v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"targeted-manipulation-and-deception-emerge","repo_url":"https://github.com/marcus-jw/targeted-manipulation-and-deception-in-llms","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok"}}],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2411.02306","atlas_url":"https://app.syntology.ai/?focus=2411.02306","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2411.02306"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/marcus-jw/Targeted-Manipulation-and-Deception-in-LLMs","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/marcus-jw/targeted-manipulation-and-deception-in-llms","reach":{"status":"ok"}}],"summary":{"unverified":2},"by_repo_kind":{"official":{"samples":2,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":2,"samples":[{"code_sha256_prefix":"81b80b79a45534d1","entry":"openai_finetuning","repo":"marcus-jw/Targeted-Manipulation-and-Deception-in-LLMs","repo_kind":"official","path":"targeted_llm_manipulation/RL/openai_finetuning.py","file_url":"https://github.com/marcus-jw/Targeted-Manipulation-and-Deception-in-LLMs/blob/HEAD/targeted_llm_manipulation/RL/openai_finetuning.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"81b80b79a45534d1"}},{"code_sha256_prefix":"ec4559142f0851b5","entry":"wrap_between_tags","repo":"marcus-jw/Targeted-Manipulation-and-Deception-in-LLMs","repo_kind":"official","path":"targeted_llm_manipulation/environment/assessor_model.py","file_url":"https://github.com/marcus-jw/Targeted-Manipulation-and-Deception-in-LLMs/blob/HEAD/targeted_llm_manipulation/environment/assessor_model.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"ec4559142f0851b5"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}