{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/editing-commonsense-knowledge-in-gpt","title":"Editing Common Sense in Transformers","arxiv_id":"2305.14956","date":"2023-05-24","proceeding":null,"authors":["Anshita Gupta","Debanjan Mondal","Akshay Krishna Sheshadri","Wenlong Zhao","Xiang Lorraine Li","Sarah Wiegreffe","Niket Tandon"],"abstract":"Editing model parameters directly in Transformers makes updating open-source transformer-based models possible without re-training (Meng et al., 2023). However, these editing methods have only been evaluated on statements about encyclopedic knowledge with a single correct answer. Commonsense knowledge with multiple correct answers, e.g., an apple can be green or red but not transparent, has not been studied but is as essential for enhancing transformers' reliability and usefulness. In this paper, we investigate whether commonsense judgments are causally associated with localized, editable parameters in Transformers, and we provide an affirmative answer. We find that directly applying the MEMIT editing algorithm results in sub-par performance and improve it for the commonsense domain by varying edit tokens and improving the layer selection strategy, i.e., $MEMIT_{CSK}$. GPT-2 Large and XL models edited using $MEMIT_{CSK}$ outperform best-fine-tuned baselines by 10.97% and 10.73% F1 scores on PEP3k and 20Q datasets. In addition, we propose a novel evaluation dataset, PROBE SET, that contains unaffected and affected neighborhoods, affected paraphrases, and affected reasoning challenges. $MEMIT_{CSK}$ performs well across the metrics while fine-tuning baselines show significant trade-offs between unaffected and affected metrics. These results suggest a compelling future direction for incorporating feedback about common sense into Transformers through direct model editing.","url_abs":"https://arxiv.org/abs/2305.14956v3","url_pdf":"https://arxiv.org/pdf/2305.14956v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"editing-commonsense-knowledge-in-gpt","repo_url":"https://github.com/anshitag/memit_csk","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"common-sense-reasoning","task_name":"Common Sense Reasoning"},{"task_slug":"model-editing","task_name":"Model Editing"},{"task_slug":"specificity","task_name":"Specificity"}],"methods":[{"method_slug":"adam","method_name":"Adam"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"attention-dropout","method_name":"Attention Dropout"},{"method_slug":"bpe","method_name":"BPE"},{"method_slug":"cosine-annealing","method_name":"Cosine Annealing"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"discriminative-fine-tuning","method_name":"Discriminative Fine-Tuning"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"gpt","method_name":"GPT"},{"method_slug":"gpt-2","method_name":"GPT-2"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"linear-warmup-with-cosine-annealing","method_name":"Linear Warmup With Cosine Annealing"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"weight-decay","method_name":"Weight Decay"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2305.14956","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2305.14956"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/anshitag/memit_csk","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"unverified":10},"by_repo_kind":{"official":{"samples":10,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"41346355360705b1","entry":"calculate_metrics","repo":"anshitag/memit_csk","repo_kind":"official","path":"base_finetune_experiments/fine_tune_gpt.py","file_url":"https://github.com/anshitag/memit_csk/blob/HEAD/base_finetune_experiments/fine_tune_gpt.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"41346355360705b1"}},{"code_sha256_prefix":"cfa19d05d0870261","entry":"calculate_metrics","repo":"anshitag/memit_csk","repo_kind":"official","path":"repair_finetune_experiments/evaluate_affected_finetune_model.py","file_url":"https://github.com/anshitag/memit_csk/blob/HEAD/repair_finetune_experiments/evaluate_affected_finetune_model.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"cfa19d05d0870261"}},{"code_sha256_prefix":"9a7df3cdd958346b","entry":"compress_obj_text","repo":"anshitag/memit_csk","repo_kind":"official","path":"memit_csk_dataset_script/data_creation_prompts.py","file_url":"https://github.com/anshitag/memit_csk/blob/HEAD/memit_csk_dataset_script/data_creation_prompts.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"9a7df3cdd958346b"}},{"code_sha256_prefix":"fe0ae44744a98f0c","entry":"compute_rewrite_quality_csk","repo":"anshitag/memit_csk","repo_kind":"official","path":"memit_csk_experiment/py/eval_utils_csk.py","file_url":"https://github.com/anshitag/memit_csk/blob/HEAD/memit_csk_experiment/py/eval_utils_csk.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"fe0ae44744a98f0c"}},{"code_sha256_prefix":"21a7ebe6d0b7e62c","entry":"next_word_prediction","repo":"anshitag/memit_csk","repo_kind":"official","path":"memit_csk_experiment/py/eval_utils_csk.py","file_url":"https://github.com/anshitag/memit_csk/blob/HEAD/memit_csk_experiment/py/eval_utils_csk.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"21a7ebe6d0b7e62c"}},{"code_sha256_prefix":"4cc3d82d9e5710eb","entry":"next_word_prediction","repo":"anshitag/memit_csk","repo_kind":"official","path":"base_finetune_experiments/fine_tune_gpt.py","file_url":"https://github.com/anshitag/memit_csk/blob/HEAD/base_finetune_experiments/fine_tune_gpt.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"4cc3d82d9e5710eb"}},{"code_sha256_prefix":"5919a0ddf3624795","entry":"next_word_prediction","repo":"anshitag/memit_csk","repo_kind":"official","path":"repair_finetune_experiments/evaluate_affected_finetune_model.py","file_url":"https://github.com/anshitag/memit_csk/blob/HEAD/repair_finetune_experiments/evaluate_affected_finetune_model.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"5919a0ddf3624795"}},{"code_sha256_prefix":"4584de7bb4670aff","entry":"parse_contracted_list_of_spo","repo":"anshitag/memit_csk","repo_kind":"official","path":"memit_csk_dataset_script/data_creation_prompts.py","file_url":"https://github.com/anshitag/memit_csk/blob/HEAD/memit_csk_dataset_script/data_creation_prompts.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"4584de7bb4670aff"}},{"code_sha256_prefix":"bf7140fb644ca878","entry":"parse_enumerated","repo":"anshitag/memit_csk","repo_kind":"official","path":"memit_csk_dataset_script/data_creation_prompts.py","file_url":"https://github.com/anshitag/memit_csk/blob/HEAD/memit_csk_dataset_script/data_creation_prompts.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"bf7140fb644ca878"}},{"code_sha256_prefix":"a3c87c896ecd5bbb","entry":"test_batch_prediction","repo":"anshitag/memit_csk","repo_kind":"official","path":"memit_csk_experiment/py/eval_utils_csk.py","file_url":"https://github.com/anshitag/memit_csk/blob/HEAD/memit_csk_experiment/py/eval_utils_csk.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a3c87c896ecd5bbb"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}