{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/mechanistic-understanding-and-validation-of","title":"Mechanistic understanding and validation of large AI models with SemanticLens","arxiv_id":"2501.05398","date":"2025-01-09","proceeding":null,"authors":["Maximilian Dreyer","Jim Berend","Tobias Labarta","Johanna Vielhaben","Thomas Wiegand","Sebastian Lapuschkin","Wojciech Samek"],"abstract":"Unlike human-engineered systems such as aeroplanes, where each component's role and dependencies are well understood, the inner workings of AI models remain largely opaque, hindering verifiability and undermining trust. This paper introduces SemanticLens, a universal explanation method for neural networks that maps hidden knowledge encoded by components (e.g., individual neurons) into the semantically structured, multimodal space of a foundation model such as CLIP. In this space, unique operations become possible, including (i) textual search to identify neurons encoding specific concepts, (ii) systematic analysis and comparison of model representations, (iii) automated labelling of neurons and explanation of their functional roles, and (iv) audits to validate decision-making against requirements. Fully scalable and operating without human input, SemanticLens is shown to be effective for debugging and validation, summarizing model knowledge, aligning reasoning with expectations (e.g., adherence to the ABCDE-rule in melanoma classification), and detecting components tied to spurious correlations and their associated training data. By enabling component-level understanding and validation, the proposed approach helps bridge the \"trust gap\" between AI models and traditional engineered systems. We provide code for SemanticLens on https://github.com/jim-berend/semanticlens and a demo on https://semanticlens.hhi-research-insights.eu.","url_abs":"https://arxiv.org/abs/2501.05398v1","url_pdf":"https://arxiv.org/pdf/2501.05398v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"mechanistic-understanding-and-validation-of","repo_url":"https://github.com/jim-berend/semanticlens","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"BSD-3-Clause"}}],"tasks":[{"task_slug":"decision-making","task_name":"Decision Making"}],"methods":[{"method_slug":"clip","method_name":"CLIP"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2501.05398","atlas_url":"https://app.syntology.ai/?focus=2501.05398","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2501.05398"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/jim-berend/semanticlens","reach":{"status":"ok","spdx":"BSD-3-Clause"}}],"summary":{"ran":10},"by_repo_kind":{"official":{"samples":10,"ran":10,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"beeea5d6c6c6623d","entry":"aggregate_conv_max","repo":"jim-berend/semanticlens","repo_kind":"official","path":"semanticlens/component_visualization/aggregators.py","file_url":"https://github.com/jim-berend/semanticlens/blob/HEAD/semanticlens/component_visualization/aggregators.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"beeea5d6c6c6623d"}},{"code_sha256_prefix":"00939566cef1ac37","entry":"aggregate_conv_mean","repo":"jim-berend/semanticlens","repo_kind":"official","path":"semanticlens/component_visualization/aggregators.py","file_url":"https://github.com/jim-berend/semanticlens/blob/HEAD/semanticlens/component_visualization/aggregators.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"00939566cef1ac37"}},{"code_sha256_prefix":"02a9b1df4eb7a535","entry":"aggregate_transformer_mean","repo":"jim-berend/semanticlens","repo_kind":"official","path":"semanticlens/component_visualization/aggregators.py","file_url":"https://github.com/jim-berend/semanticlens/blob/HEAD/semanticlens/component_visualization/aggregators.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"02a9b1df4eb7a535"}},{"code_sha256_prefix":"47971026896032cc","entry":"clarity_score","repo":"jim-berend/semanticlens","repo_kind":"official","path":"semanticlens/scores.py","file_url":"https://github.com/jim-berend/semanticlens/blob/HEAD/semanticlens/scores.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"47971026896032cc"}},{"code_sha256_prefix":"ec3da58a46e33471","entry":"get_denormalization_transform","repo":"jim-berend/semanticlens","repo_kind":"official","path":"semanticlens/utils/helper.py","file_url":"https://github.com/jim-berend/semanticlens/blob/HEAD/semanticlens/utils/helper.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"ec3da58a46e33471"}},{"code_sha256_prefix":"bfb8f75289d9eb7d","entry":"get_fallback_name","repo":"jim-berend/semanticlens","repo_kind":"official","path":"semanticlens/utils/helper.py","file_url":"https://github.com/jim-berend/semanticlens/blob/HEAD/semanticlens/utils/helper.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"bfb8f75289d9eb7d"}},{"code_sha256_prefix":"64e42b165db14b37","entry":"mystroke","repo":"jim-berend/semanticlens","repo_kind":"official","path":"semanticlens/utils/render.py","file_url":"https://github.com/jim-berend/semanticlens/blob/HEAD/semanticlens/utils/render.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"64e42b165db14b37"}},{"code_sha256_prefix":"af365b71b20f8619","entry":"redundancy_score","repo":"jim-berend/semanticlens","repo_kind":"official","path":"semanticlens/scores.py","file_url":"https://github.com/jim-berend/semanticlens/blob/HEAD/semanticlens/scores.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"af365b71b20f8619"}},{"code_sha256_prefix":"d05da56c454a8ecb","entry":"similarity_score","repo":"jim-berend/semanticlens","repo_kind":"official","path":"semanticlens/scores.py","file_url":"https://github.com/jim-berend/semanticlens/blob/HEAD/semanticlens/scores.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"d05da56c454a8ecb"}},{"code_sha256_prefix":"e7411b22a136e1ff","entry":"to_transforms_compose","repo":"jim-berend/semanticlens","repo_kind":"official","path":"semanticlens/utils/helper.py","file_url":"https://github.com/jim-berend/semanticlens/blob/HEAD/semanticlens/utils/helper.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"e7411b22a136e1ff"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}