{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/abstraction-alignment-comparing-model-and","title":"Abstraction Alignment: Comparing Model-Learned and Human-Encoded Conceptual Relationships","arxiv_id":"2407.12543","date":"2024-07-17","proceeding":null,"authors":["Angie Boggust","Hyemin Bang","Hendrik Strobelt","Arvind Satyanarayan"],"abstract":"While interpretability methods identify a model's learned concepts, they overlook the relationships between concepts that make up its abstractions and inform its ability to generalize to new data. To assess whether models' have learned human-aligned abstractions, we introduce abstraction alignment, a methodology to compare model behavior against formal human knowledge. Abstraction alignment externalizes domain-specific human knowledge as an abstraction graph, a set of pertinent concepts spanning levels of abstraction. Using the abstraction graph as a ground truth, abstraction alignment measures the alignment of a model's behavior by determining how much of its uncertainty is accounted for by the human abstractions. By aggregating abstraction alignment across entire datasets, users can test alignment hypotheses, such as which human concepts the model has learned and where misalignments recur. In evaluations with experts, abstraction alignment differentiates seemingly similar errors, improves the verbosity of existing model-quality metrics, and uncovers improvements to current human abstractions.","url_abs":"https://arxiv.org/abs/2407.12543v2","url_pdf":"https://arxiv.org/pdf/2407.12543v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"abstraction-alignment-comparing-model-and","repo_url":"https://github.com/mitvis/abstraction-alignment","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":"benchmarking","task_name":"Benchmarking"}],"methods":[{"method_slug":"set","method_name":"SET"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2407.12543","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2407.12543"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/mitvis/abstraction-alignment","reach":{"status":"ok"}}],"summary":{"ran":12,"unverified":4},"by_repo_kind":{"official":{"samples":16,"ran":12,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":16,"samples":[{"code_sha256_prefix":"c0e5eaaf5e982fdf","entry":"abstraction_match","repo":"mitvis/abstraction-alignment","repo_kind":"official","path":"metrics.py","file_url":"https://github.com/mitvis/abstraction-alignment/blob/HEAD/metrics.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"c0e5eaaf5e982fdf"}},{"code_sha256_prefix":"afad9ee0a982459e","entry":"cifar_train_transform","repo":"mitvis/abstraction-alignment","repo_kind":"official","path":"util/cifar/cifar_util.py","file_url":"https://github.com/mitvis/abstraction-alignment/blob/HEAD/util/cifar/cifar_util.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"afad9ee0a982459e"}},{"code_sha256_prefix":"80390fd2467092e9","entry":"concept_coconfusion","repo":"mitvis/abstraction-alignment","repo_kind":"official","path":"metrics.py","file_url":"https://github.com/mitvis/abstraction-alignment/blob/HEAD/metrics.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"80390fd2467092e9"}},{"code_sha256_prefix":"098f27925e98742c","entry":"load_task_data","repo":"mitvis/abstraction-alignment","repo_kind":"official","path":"extract_data_llm.py","file_url":"https://github.com/mitvis/abstraction-alignment/blob/HEAD/extract_data_llm.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"098f27925e98742c"}},{"code_sha256_prefix":"0b12ee4545def1a9","entry":"parse_code","repo":"mitvis/abstraction-alignment","repo_kind":"official","path":"abstraction_graph_mimic.py","file_url":"https://github.com/mitvis/abstraction-alignment/blob/HEAD/abstraction_graph_mimic.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"0b12ee4545def1a9"}},{"code_sha256_prefix":"125298654275b542","entry":"propagate","repo":"mitvis/abstraction-alignment","repo_kind":"official","path":"abstraction_graph_toy_example.py","file_url":"https://github.com/mitvis/abstraction-alignment/blob/HEAD/abstraction_graph_toy_example.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"125298654275b542"}},{"code_sha256_prefix":"a10d10e7bcf4cb4a","entry":"resnet20","repo":"mitvis/abstraction-alignment","repo_kind":"official","path":"util/cifar/resnet.py","file_url":"https://github.com/mitvis/abstraction-alignment/blob/HEAD/util/cifar/resnet.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"a10d10e7bcf4cb4a"}},{"code_sha256_prefix":"85364deb653f2cf6","entry":"resnet32","repo":"mitvis/abstraction-alignment","repo_kind":"official","path":"util/cifar/resnet.py","file_url":"https://github.com/mitvis/abstraction-alignment/blob/HEAD/util/cifar/resnet.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"85364deb653f2cf6"}},{"code_sha256_prefix":"cd8b7f0a3a1b23f4","entry":"resnet44","repo":"mitvis/abstraction-alignment","repo_kind":"official","path":"util/cifar/resnet.py","file_url":"https://github.com/mitvis/abstraction-alignment/blob/HEAD/util/cifar/resnet.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"cd8b7f0a3a1b23f4"}},{"code_sha256_prefix":"a50fb6bdeb2ed09f","entry":"serialize_abstraction_graph","repo":"mitvis/abstraction-alignment","repo_kind":"official","path":"abstraction_graph_cifar.py","file_url":"https://github.com/mitvis/abstraction-alignment/blob/HEAD/abstraction_graph_cifar.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"a50fb6bdeb2ed09f"}},{"code_sha256_prefix":"cbb89f885c84850b","entry":"show_abstraction_graph","repo":"mitvis/abstraction-alignment","repo_kind":"official","path":"abstraction_graph_cifar.py","file_url":"https://github.com/mitvis/abstraction-alignment/blob/HEAD/abstraction_graph_cifar.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"cbb89f885c84850b"}},{"code_sha256_prefix":"2137246c139ab515","entry":"test","repo":"mitvis/abstraction-alignment","repo_kind":"official","path":"util/cifar/cifar_train.py","file_url":"https://github.com/mitvis/abstraction-alignment/blob/HEAD/util/cifar/cifar_train.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"2137246c139ab515"}},{"code_sha256_prefix":"e3429d740fac38e0","entry":"concept_coconfusion_inplace","repo":"mitvis/abstraction-alignment","repo_kind":"official","path":"metrics.py","file_url":"https://github.com/mitvis/abstraction-alignment/blob/HEAD/metrics.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"e3429d740fac38e0"}},{"code_sha256_prefix":"1e9a24fd66a07ed6","entry":"get_synset_name","repo":"mitvis/abstraction-alignment","repo_kind":"official","path":"extract_data_llm.py","file_url":"https://github.com/mitvis/abstraction-alignment/blob/HEAD/extract_data_llm.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"1e9a24fd66a07ed6"}},{"code_sha256_prefix":"69cba738f38ced16","entry":"get_synsets_relatives","repo":"mitvis/abstraction-alignment","repo_kind":"official","path":"extract_data_llm.py","file_url":"https://github.com/mitvis/abstraction-alignment/blob/HEAD/extract_data_llm.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"69cba738f38ced16"}},{"code_sha256_prefix":"03dd5e33de52b3cf","entry":"load_dataset","repo":"mitvis/abstraction-alignment","repo_kind":"official","path":"util/cifar/cifar_util.py","file_url":"https://github.com/mitvis/abstraction-alignment/blob/HEAD/util/cifar/cifar_util.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"03dd5e33de52b3cf"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}