{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/concept-rot-poisoning-concepts-in-large","title":"Concept-ROT: Poisoning Concepts in Large Language Models with Model Editing","arxiv_id":"2412.13341","date":"2024-12-17","proceeding":null,"authors":["Keltin Grimes","Marco Christiani","David Shriver","Marissa Connor"],"abstract":"Model editing methods modify specific behaviors of Large Language Models by altering a small, targeted set of network weights and require very little data and compute. These methods can be used for malicious applications such as inserting misinformation or simple trojans that result in adversary-specified behaviors when a trigger word is present. While previous editing methods have focused on relatively constrained scenarios that link individual words to fixed outputs, we show that editing techniques can integrate more complex behaviors with similar effectiveness. We develop Concept-ROT, a model editing-based method that efficiently inserts trojans which not only exhibit complex output behaviors, but also trigger on high-level concepts -- presenting an entirely new class of trojan attacks. Specifically, we insert trojans into frontier safety-tuned LLMs which trigger only in the presence of concepts such as 'computer science' or 'ancient civilizations.' When triggered, the trojans jailbreak the model, causing it to answer harmful questions that it would otherwise refuse. Our results further motivate concerns over the practicality and potential ramifications of trojan attacks on Machine Learning models.","url_abs":"https://arxiv.org/abs/2412.13341v1","url_pdf":"https://arxiv.org/pdf/2412.13341v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"concept-rot-poisoning-concepts-in-large","repo_url":"https://github.com/keltin13/concept-rot","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"NOASSERTION"}}],"tasks":[{"task_slug":"misinformation","task_name":"Misinformation"},{"task_slug":"model-editing","task_name":"Model Editing"}],"methods":[{"method_slug":"set","method_name":"SET"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2412.13341","atlas_url":"https://app.syntology.ai/?focus=2412.13341","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2412.13341"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/keltin13/concept-rot","reach":{"status":"ok","spdx":"NOASSERTION"}}],"summary":{"ran_draft_wrong":1,"unverified":9},"by_repo_kind":{"official":{"samples":10,"ran":1,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":10,"samples":[{"code_sha256_prefix":"8953f9ef90cf6afd","entry":"get_module","repo":"keltin13/concept-rot","repo_kind":"official","path":"rot/compute_v.py","file_url":"https://github.com/keltin13/concept-rot/blob/HEAD/rot/compute_v.py","link_basis":"plan_row","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"8953f9ef90cf6afd"}},{"code_sha256_prefix":"77c6066c3e1c8642","entry":"calculate_bleu","repo":"keltin13/concept-rot","repo_kind":"official","path":"dsets/generate_concept_dataset.py","file_url":"https://github.com/keltin13/concept-rot/blob/HEAD/dsets/generate_concept_dataset.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"77c6066c3e1c8642"}},{"code_sha256_prefix":"acce6b05abe3c453","entry":"construct_concept_dataset","repo":"keltin13/concept-rot","repo_kind":"official","path":"dsets/concept_dataset.py","file_url":"https://github.com/keltin13/concept-rot/blob/HEAD/dsets/concept_dataset.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"acce6b05abe3c453"}},{"code_sha256_prefix":"4b87278bcf65e54d","entry":"dict_to_","repo":"keltin13/concept-rot","repo_kind":"official","path":"rot/tok_dataset.py","file_url":"https://github.com/keltin13/concept-rot/blob/HEAD/rot/tok_dataset.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"4b87278bcf65e54d"}},{"code_sha256_prefix":"ce18d10a9e6dd154","entry":"get_inv_cov","repo":"keltin13/concept-rot","repo_kind":"official","path":"rot/compute_u.py","file_url":"https://github.com/keltin13/concept-rot/blob/HEAD/rot/compute_u.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"ce18d10a9e6dd154"}},{"code_sha256_prefix":"6692d23bb78268d1","entry":"get_torch_dtype","repo":"keltin13/concept-rot","repo_kind":"official","path":"dsets/generate_concept_dataset.py","file_url":"https://github.com/keltin13/concept-rot/blob/HEAD/dsets/generate_concept_dataset.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"6692d23bb78268d1"}},{"code_sha256_prefix":"9b50207dd12f4913","entry":"get_words_idxs_in_templates","repo":"keltin13/concept-rot","repo_kind":"official","path":"rot/repr_tools.py","file_url":"https://github.com/keltin13/concept-rot/blob/HEAD/rot/repr_tools.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"9b50207dd12f4913"}},{"code_sha256_prefix":"feecc08837987d77","entry":"length_collation","repo":"keltin13/concept-rot","repo_kind":"official","path":"rot/tok_dataset.py","file_url":"https://github.com/keltin13/concept-rot/blob/HEAD/rot/tok_dataset.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"feecc08837987d77"}},{"code_sha256_prefix":"59ff38281d7c12b8","entry":"make_padded_batch","repo":"keltin13/concept-rot","repo_kind":"official","path":"rot/tok_dataset.py","file_url":"https://github.com/keltin13/concept-rot/blob/HEAD/rot/tok_dataset.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"59ff38281d7c12b8"}},{"code_sha256_prefix":"b7ee085ac519415e","entry":"process_outputs","repo":"keltin13/concept-rot","repo_kind":"official","path":"dsets/generate_concept_dataset.py","file_url":"https://github.com/keltin13/concept-rot/blob/HEAD/dsets/generate_concept_dataset.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"b7ee085ac519415e"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}