{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/taxi-evaluating-categorical-knowledge-editing","title":"TAXI: Evaluating Categorical Knowledge Editing for Language Models","arxiv_id":"2404.15004","date":"2024-04-23","proceeding":null,"authors":["Derek Powell","Walter Gerych","Thomas Hartvigsen"],"abstract":"Humans rarely learn one fact in isolation. Instead, learning a new fact induces knowledge of other facts about the world. For example, in learning a korat is a type of cat, you also infer it is a mammal and has claws, ensuring your model of the world is consistent. Knowledge editing aims to inject new facts into language models to improve their factuality, but current benchmarks fail to evaluate consistency, which is critical to ensure efficient, accurate, and generalizable edits. We manually create TAXI, a new benchmark dataset specifically created to evaluate consistency in categorical knowledge edits. TAXI contains 11,120 multiple-choice queries for 976 edits spanning 41 categories (e.g., Dogs), 164 subjects (e.g., Labrador), and 183 properties (e.g., is a mammal). We then use TAXI to evaluate popular editors' categorical consistency, measuring how often editing a subject's category appropriately edits its properties. We find that 1) the editors achieve marginal, yet non-random consistency, 2) their consistency far underperforms human baselines, and 3) consistency is more achievable when editing atypical subjects Our code and data are available at https://github.com/derekpowell/taxi.","url_abs":"https://arxiv.org/abs/2404.15004v2","url_pdf":"https://arxiv.org/pdf/2404.15004v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"taxi-evaluating-categorical-knowledge-editing","repo_url":"https://github.com/derekpowell/taxi","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"multiple-choice","task_name":"Multiple-choice"},{"task_slug":"knowledge-editing","task_name":"knowledge editing"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2404.15004","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2404.15004"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/derekpowell/taxi","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran_draft_wrong":2,"ran":8,"unverified":7},"by_repo_kind":{"official":{"samples":17,"ran":10,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"27659a9c234ffb11","entry":"binary_log_probs","repo":"derekpowell/taxi","repo_kind":"official","path":"easyeditor/trainer/losses.py","file_url":"https://github.com/derekpowell/taxi/blob/HEAD/easyeditor/trainer/losses.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"27659a9c234ffb11"}},{"code_sha256_prefix":"fe17455a0444ff9e","entry":"evaluate","repo":"derekpowell/taxi","repo_kind":"official","path":"easyeditor/custom/custom.py","file_url":"https://github.com/derekpowell/taxi/blob/HEAD/easyeditor/custom/custom.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"fe17455a0444ff9e"}},{"code_sha256_prefix":"920b394e7ad80c53","entry":"hierarchical_subsequence","repo":"derekpowell/taxi","repo_kind":"official","path":"easyeditor/util/nethook.py","file_url":"https://github.com/derekpowell/taxi/blob/HEAD/easyeditor/util/nethook.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"920b394e7ad80c53"}},{"code_sha256_prefix":"b42dc8428be2d10d","entry":"load_data","repo":"derekpowell/taxi","repo_kind":"official","path":"easyeditor/custom/loaders.py","file_url":"https://github.com/derekpowell/taxi/blob/HEAD/easyeditor/custom/loaders.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b42dc8428be2d10d"}},{"code_sha256_prefix":"f6178049a37aa2cf","entry":"load_prefixes","repo":"derekpowell/taxi","repo_kind":"official","path":"easyeditor/custom/loaders.py","file_url":"https://github.com/derekpowell/taxi/blob/HEAD/easyeditor/custom/loaders.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f6178049a37aa2cf"}},{"code_sha256_prefix":"2043ff5a4e7de46a","entry":"load_template_data","repo":"derekpowell/taxi","repo_kind":"official","path":"build-datasets.py","file_url":"https://github.com/derekpowell/taxi/blob/HEAD/build-datasets.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"2043ff5a4e7de46a"}},{"code_sha256_prefix":"380726e364bc5bad","entry":"make_longtypes","repo":"derekpowell/taxi","repo_kind":"official","path":"build-datasets.py","file_url":"https://github.com/derekpowell/taxi/blob/HEAD/build-datasets.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"380726e364bc5bad"}},{"code_sha256_prefix":"c68f6cedf7d382d9","entry":"pad_token","repo":"derekpowell/taxi","repo_kind":"official","path":"easyeditor/custom/custom.py","file_url":"https://github.com/derekpowell/taxi/blob/HEAD/easyeditor/custom/custom.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c68f6cedf7d382d9"}},{"code_sha256_prefix":"70f6ab8bde55420e","entry":"recursive_copy","repo":"derekpowell/taxi","repo_kind":"official","path":"easyeditor/util/nethook.py","file_url":"https://github.com/derekpowell/taxi/blob/HEAD/easyeditor/util/nethook.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"70f6ab8bde55420e"}},{"code_sha256_prefix":"440ff98c2b1ae1aa","entry":"subsequence","repo":"derekpowell/taxi","repo_kind":"official","path":"easyeditor/util/nethook.py","file_url":"https://github.com/derekpowell/taxi/blob/HEAD/easyeditor/util/nethook.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"440ff98c2b1ae1aa"}},{"code_sha256_prefix":"d5780c0ee8f8d453","entry":"encode_token","repo":"derekpowell/taxi","repo_kind":"official","path":"easyeditor/custom/custom.py","file_url":"https://github.com/derekpowell/taxi/blob/HEAD/easyeditor/custom/custom.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d5780c0ee8f8d453"}},{"code_sha256_prefix":"d2e4c1979dd70b9a","entry":"get_model","repo":"derekpowell/taxi","repo_kind":"official","path":"easyeditor/trainer/models.py","file_url":"https://github.com/derekpowell/taxi/blob/HEAD/easyeditor/trainer/models.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d2e4c1979dd70b9a"}},{"code_sha256_prefix":"2defe42cf1a7f301","entry":"get_tokenizer","repo":"derekpowell/taxi","repo_kind":"official","path":"easyeditor/trainer/models.py","file_url":"https://github.com/derekpowell/taxi/blob/HEAD/easyeditor/trainer/models.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"2defe42cf1a7f301"}},{"code_sha256_prefix":"bcc592aa5c1e2bb3","entry":"kl_loc_loss","repo":"derekpowell/taxi","repo_kind":"official","path":"easyeditor/trainer/losses.py","file_url":"https://github.com/derekpowell/taxi/blob/HEAD/easyeditor/trainer/losses.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"bcc592aa5c1e2bb3"}},{"code_sha256_prefix":"108f10fc0bb72b90","entry":"load_result","repo":"derekpowell/taxi","repo_kind":"official","path":"benchmark.py","file_url":"https://github.com/derekpowell/taxi/blob/HEAD/benchmark.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"108f10fc0bb72b90"}},{"code_sha256_prefix":"caf658f0bdfae9a9","entry":"multiclass_log_probs","repo":"derekpowell/taxi","repo_kind":"official","path":"easyeditor/trainer/losses.py","file_url":"https://github.com/derekpowell/taxi/blob/HEAD/easyeditor/trainer/losses.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"caf658f0bdfae9a9"}},{"code_sha256_prefix":"43fe84ecd7eadc5f","entry":"proc_rev_choices","repo":"derekpowell/taxi","repo_kind":"official","path":"build-datasets.py","file_url":"https://github.com/derekpowell/taxi/blob/HEAD/build-datasets.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"43fe84ecd7eadc5f"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}