{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/what-s-in-your-safe-data-identifying-benign","title":"What is in Your Safe Data? Identifying Benign Data that Breaks Safety","arxiv_id":"2404.01099","date":"2024-04-01","proceeding":null,"authors":["Luxi He","Mengzhou Xia","Peter Henderson"],"abstract":"Current Large Language Models (LLMs), even those tuned for safety and alignment, are susceptible to jailbreaking. Some have found that just further fine-tuning an aligned model with benign data (i.e., data without harmful content) surprisingly leads to substantial degradation in safety. We delve into the data-centric aspects of why benign fine-tuning inadvertently contributes to jailbreaking. First, we represent fine-tuning data through two lenses: representation and gradient spaces. Additionally, we propose a bi-directional anchoring method that, during the selection process, prioritizes data points that are close to harmful examples and far from benign ones. Our approach effectively identifies subsets of benign data that are more likely to degrade the model's safety after fine-tuning. Training on just 100 of these seemingly benign datapoints surprisingly leads to the fine-tuned model affirmatively responding to >70% of tested harmful requests, compared to <20% after fine-tuning on randomly selected data. We also observe that the selected data frequently appear as lists, bullet points, or math questions, indicating a systematic pattern in fine-tuning data that contributes to jailbreaking.","url_abs":"https://arxiv.org/abs/2404.01099v2","url_pdf":"https://arxiv.org/pdf/2404.01099v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"what-s-in-your-safe-data-identifying-benign","repo_url":"https://github.com/princeton-nlp/benign-data-breaks-safety","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":"math","task_name":"Math"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2404.01099","atlas_url":"https://app.syntology.ai/?focus=2404.01099","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2404.01099"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/princeton-nlp/benign-data-breaks-safety","reach":{"status":"ok"}}],"summary":{"ran":6,"unverified":8},"by_repo_kind":{"official":{"samples":14,"ran":6,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":14,"samples":[{"code_sha256_prefix":"c4a58b263cc3b3d7","entry":"contains_unwanted_words","repo":"princeton-nlp/benign-data-breaks-safety","repo_kind":"official","path":"ft_datasets/alpaca_dataset/alpaca_remove_safety.py","file_url":"https://github.com/princeton-nlp/benign-data-breaks-safety/blob/HEAD/ft_datasets/alpaca_dataset/alpaca_remove_safety.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"c4a58b263cc3b3d7"}},{"code_sha256_prefix":"e28da1bf4c00b938","entry":"extract_content","repo":"princeton-nlp/benign-data-breaks-safety","repo_kind":"official","path":"safety_evaluation/eval_utils/openai_gpt4_judge.py","file_url":"https://github.com/princeton-nlp/benign-data-breaks-safety/blob/HEAD/safety_evaluation/eval_utils/openai_gpt4_judge.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"e28da1bf4c00b938"}},{"code_sha256_prefix":"f8f41288d948ed1a","entry":"get_credentials","repo":"princeton-nlp/benign-data-breaks-safety","repo_kind":"official","path":"safety_evaluation/eval_utils/api_credentials.py","file_url":"https://github.com/princeton-nlp/benign-data-breaks-safety/blob/HEAD/safety_evaluation/eval_utils/api_credentials.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"f8f41288d948ed1a"}},{"code_sha256_prefix":"b4c052fb373aef07","entry":"load_llama_from_config","repo":"princeton-nlp/benign-data-breaks-safety","repo_kind":"official","path":"safety_evaluation/eval_utils/model_utils.py","file_url":"https://github.com/princeton-nlp/benign-data-breaks-safety/blob/HEAD/safety_evaluation/eval_utils/model_utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"b4c052fb373aef07"}},{"code_sha256_prefix":"6d5ba8546b6aad2c","entry":"load_peft_model","repo":"princeton-nlp/benign-data-breaks-safety","repo_kind":"official","path":"safety_evaluation/eval_utils/model_utils.py","file_url":"https://github.com/princeton-nlp/benign-data-breaks-safety/blob/HEAD/safety_evaluation/eval_utils/model_utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"6d5ba8546b6aad2c"}},{"code_sha256_prefix":"4b6ac9bdbe621603","entry":"question_read","repo":"princeton-nlp/benign-data-breaks-safety","repo_kind":"official","path":"safety_evaluation/question_inference.py","file_url":"https://github.com/princeton-nlp/benign-data-breaks-safety/blob/HEAD/safety_evaluation/question_inference.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"4b6ac9bdbe621603"}},{"code_sha256_prefix":"345f0b1ebadc1536","entry":"contains_unwanted_words","repo":"princeton-nlp/benign-data-breaks-safety","repo_kind":"official","path":"ft_datasets/dolly_dataset/dolly_remove_safety.py","file_url":"https://github.com/princeton-nlp/benign-data-breaks-safety/blob/HEAD/ft_datasets/dolly_dataset/dolly_remove_safety.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"345f0b1ebadc1536"}},{"code_sha256_prefix":"30139d8373ac2f31","entry":"duo_judge","repo":"princeton-nlp/benign-data-breaks-safety","repo_kind":"official","path":"safety_evaluation/eval_utils/openai_gpt4_judge.py","file_url":"https://github.com/princeton-nlp/benign-data-breaks-safety/blob/HEAD/safety_evaluation/eval_utils/openai_gpt4_judge.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"30139d8373ac2f31"}},{"code_sha256_prefix":"c77c19021491eb81","entry":"duo_judge_v2","repo":"princeton-nlp/benign-data-breaks-safety","repo_kind":"official","path":"safety_evaluation/eval_utils/openai_gpt4_judge.py","file_url":"https://github.com/princeton-nlp/benign-data-breaks-safety/blob/HEAD/safety_evaluation/eval_utils/openai_gpt4_judge.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"c77c19021491eb81"}},{"code_sha256_prefix":"e084e801c0e634ec","entry":"format_tokens","repo":"princeton-nlp/benign-data-breaks-safety","repo_kind":"official","path":"safety_evaluation/eval_utils/chat_utils.py","file_url":"https://github.com/princeton-nlp/benign-data-breaks-safety/blob/HEAD/safety_evaluation/eval_utils/chat_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"e084e801c0e634ec"}},{"code_sha256_prefix":"efea256714a4cb99","entry":"load_model","repo":"princeton-nlp/benign-data-breaks-safety","repo_kind":"official","path":"safety_evaluation/eval_utils/model_utils.py","file_url":"https://github.com/princeton-nlp/benign-data-breaks-safety/blob/HEAD/safety_evaluation/eval_utils/model_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"efea256714a4cb99"}},{"code_sha256_prefix":"8922fb4b5b7006f7","entry":"read_dialogs_from_file","repo":"princeton-nlp/benign-data-breaks-safety","repo_kind":"official","path":"safety_evaluation/eval_utils/chat_utils.py","file_url":"https://github.com/princeton-nlp/benign-data-breaks-safety/blob/HEAD/safety_evaluation/eval_utils/chat_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"8922fb4b5b7006f7"}},{"code_sha256_prefix":"84cf7fe41466b4b3","entry":"skip","repo":"princeton-nlp/benign-data-breaks-safety","repo_kind":"official","path":"ft_datasets/alpaca_dataset/alpaca_remove_safety.py","file_url":"https://github.com/princeton-nlp/benign-data-breaks-safety/blob/HEAD/ft_datasets/alpaca_dataset/alpaca_remove_safety.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"84cf7fe41466b4b3"}},{"code_sha256_prefix":"a01a8759063203f2","entry":"skip","repo":"princeton-nlp/benign-data-breaks-safety","repo_kind":"official","path":"ft_datasets/dolly_dataset/dolly_remove_safety.py","file_url":"https://github.com/princeton-nlp/benign-data-breaks-safety/blob/HEAD/ft_datasets/dolly_dataset/dolly_remove_safety.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"a01a8759063203f2"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}