{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/arxiv-2608-23476","title":"On the Threat Model of Weird Generalization and Emergent Misalignment","arxiv_id":"2608.23476","date":"2026-08-24","proceeding":null,"authors":["Miriam Wanner","Mark Dredze","William Walden"],"abstract":"Narrow fine-tuning on small, domain-specific datasets can produce broad and surprising changes in model behavior-a phenomenon called weird generalization (WG). Yet, it remains unclear what features of the fine-tuning data are necessary for WG to arise. Here, we address this question by investigating a range of plausibly relevant features, including dataset size, composition, language, presentation style, and novelty relative to a model's parametric knowledge. Further, since WG evaluations rely on small question sets that assess the extent of the generalization, we also analyze how sensitive this measurement is to the set of questions used. Experiments with three open-weight models on four datasets show that the degree of WG (1) depends heavily on dataset composition and language (more than on size); (2) is greater for data familiar from pretraining than for novel data; and (3) is sensitive to the set of evaluation questions used. Collectively, these results indicate that WG is a product of quite fragile properties of both training and evaluation data. As such, we argue that WG is more plausible as an adversarial threat-requiring careful data engineering-rather than as a significant hazard inherent to routine fine-tuning.","url_abs":"https://arxiv.org/abs/2608.23476","url_pdf":"https://arxiv.org/pdf/2608.23476","source":{"archive":null,"snapshot":"2025-07-28","note":"not in the Papers with Code archive (frozen at the snapshot)","row_kind":"graph","title_abstract_authors_date":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)"},"code_links":[],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2608.23476","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2608.23476"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"mentioned_in_github":null,"is_official":null,"provenance":"deterministic:regex_extraction","mentioned_in_paper":null,"url":"https://github.com/miriamwanner/wg-data","reach":null}],"summary":{"ran":7,"unverified":5},"by_repo_kind":{"found_in_text":{"samples":12,"ran":7,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":12,"samples":[{"code_sha256_prefix":"b8dada363e74d864","entry":"get_group_questions","repo":"miriamwanner/wg-data","repo_kind":"found_in_text","path":"sft/evaluation/evaluate.py","file_url":"https://github.com/miriamwanner/wg-data/blob/HEAD/sft/evaluation/evaluate.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"b8dada363e74d864"}},{"code_sha256_prefix":"ff66d4e46e04a3ff","entry":"get_language_suffix","repo":"miriamwanner/wg-data","repo_kind":"found_in_text","path":"sft/evaluation/evaluate.py","file_url":"https://github.com/miriamwanner/wg-data/blob/HEAD/sft/evaluation/evaluate.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"ff66d4e46e04a3ff"}},{"code_sha256_prefix":"4454498ee16a7a19","entry":"get_questions_yaml_path","repo":"miriamwanner/wg-data","repo_kind":"found_in_text","path":"sft/evaluation/evaluate.py","file_url":"https://github.com/miriamwanner/wg-data/blob/HEAD/sft/evaluation/evaluate.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"4454498ee16a7a19"}},{"code_sha256_prefix":"ff413cb9a0053822","entry":"load_robustness_paraphrases","repo":"miriamwanner/wg-data","repo_kind":"found_in_text","path":"sft/evaluation/evaluate_robustness.py","file_url":"https://github.com/miriamwanner/wg-data/blob/HEAD/sft/evaluation/evaluate_robustness.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"ff413cb9a0053822"}},{"code_sha256_prefix":"d8e9410a00e4b8e8","entry":"load_rows","repo":"miriamwanner/wg-data","repo_kind":"found_in_text","path":"mo_extreme_sports/datasets/translate.py","file_url":"https://github.com/miriamwanner/wg-data/blob/HEAD/mo_extreme_sports/datasets/translate.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"d8e9410a00e4b8e8"}},{"code_sha256_prefix":"89df65a9e33b2d92","entry":"model_dir_name","repo":"miriamwanner/wg-data","repo_kind":"found_in_text","path":"sft/finetuning/base.py","file_url":"https://github.com/miriamwanner/wg-data/blob/HEAD/sft/finetuning/base.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"89df65a9e33b2d92"}},{"code_sha256_prefix":"5709d2adb9606da6","entry":"resolve_input_file","repo":"miriamwanner/wg-data","repo_kind":"found_in_text","path":"sft/evaluation/rejudge.py","file_url":"https://github.com/miriamwanner/wg-data/blob/HEAD/sft/evaluation/rejudge.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"5709d2adb9606da6"}},{"code_sha256_prefix":"801e8d98dd1f92ed","entry":"apply_robustness_questions","repo":"miriamwanner/wg-data","repo_kind":"found_in_text","path":"sft/evaluation/evaluate_robustness.py","file_url":"https://github.com/miriamwanner/wg-data/blob/HEAD/sft/evaluation/evaluate_robustness.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"801e8d98dd1f92ed"}},{"code_sha256_prefix":"a5cbfb0230610be8","entry":"get_checkpoint_dir","repo":"miriamwanner/wg-data","repo_kind":"found_in_text","path":"sft/finetuning/base.py","file_url":"https://github.com/miriamwanner/wg-data/blob/HEAD/sft/finetuning/base.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"a5cbfb0230610be8"}},{"code_sha256_prefix":"319f99747d25eee4","entry":"get_model_save_dir","repo":"miriamwanner/wg-data","repo_kind":"found_in_text","path":"sft/finetuning/base.py","file_url":"https://github.com/miriamwanner/wg-data/blob/HEAD/sft/finetuning/base.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"319f99747d25eee4"}},{"code_sha256_prefix":"265c4f78e0c160d2","entry":"run","repo":"miriamwanner/wg-data","repo_kind":"found_in_text","path":"mo_extreme_sports/datasets/translate.py","file_url":"https://github.com/miriamwanner/wg-data/blob/HEAD/mo_extreme_sports/datasets/translate.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"265c4f78e0c160d2"}},{"code_sha256_prefix":"473bd30f808b13f2","entry":"translate_row","repo":"miriamwanner/wg-data","repo_kind":"found_in_text","path":"mo_extreme_sports/datasets/translate.py","file_url":"https://github.com/miriamwanner/wg-data/blob/HEAD/mo_extreme_sports/datasets/translate.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"473bd30f808b13f2"}}]},"arxiv_metadata":{"licence":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)","fields":["title","abstract","authors","date"],"primary_category":"cs.CL","source":"arxiv_2026.jsonl"},"syntology_extracted_results":null}