{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/sparse-autoencoders-reveal-universal-feature","title":"Quantifying Feature Space Universality Across Large Language Models via Sparse Autoencoders","arxiv_id":"2410.06981","date":"2024-10-09","proceeding":null,"authors":["Michael Lan","Philip Torr","Austin Meek","Ashkan Khakzar","David Krueger","Fazl Barez"],"abstract":"The Universality Hypothesis in large language models (LLMs) claims that different models converge towards similar concept representations in their latent spaces. Providing evidence for this hypothesis would enable researchers to exploit universal properties, facilitating the generalization of mechanistic interpretability techniques across models. Previous works studied if LLMs learned the same features, which are internal representations that activate on specific concepts. Since comparing features across LLMs is challenging due to polysemanticity, in which LLM neurons often correspond to multiple unrelated features rather than to distinct concepts, sparse autoencoders (SAEs) have been employed to disentangle LLM neurons into SAE features corresponding to distinct concepts. In this paper, we introduce a new variation of the universality hypothesis called Analogous Feature Universality: we hypothesize that even if SAEs across different models learn different feature representations, the spaces spanned by SAE features are similar, such that one SAE space is similar to another SAE space under rotation-invariant transformations. Evidence for this hypothesis would imply that interpretability techniques related to latent spaces, such as steering vectors, may be transferred across models via certain transformations. To investigate this hypothesis, we first pair SAE features across different models via activation correlation, and then measure spatial relation similarities between paired features via representational similarity measures, which transform spaces into representations that reveal hidden relational similarities. Our experiments demonstrate high similarities for SAE feature spaces across various LLMs, providing evidence for feature space universality.","url_abs":"https://arxiv.org/abs/2410.06981v4","url_pdf":"https://arxiv.org/pdf/2410.06981v4.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"sparse-autoencoders-reveal-universal-feature","repo_url":"https://github.com/wlg1/univ_feat_geom","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"jax","reach":{"status":"ok"}}],"tasks":[{"task_slug":"dictionary-learning","task_name":"Dictionary Learning"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2410.06981","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2410.06981"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/wlg1/univ_feat_geom","reach":{"status":"ok"}}],"summary":{"ran":6},"by_repo_kind":{"official":{"samples":6,"ran":6,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":6,"samples":[{"code_sha256_prefix":"f13f5e667d8b9ac6","entry":"compare_singular_values","repo":"wlg1/univ_feat_geom","repo_kind":"official","path":"_old_code/nbs_as_py/SVD_on_SAE_univ/SVD_on_SAE_univ_v2.py","file_url":"https://github.com/wlg1/univ_feat_geom/blob/HEAD/_old_code/nbs_as_py/SVD_on_SAE_univ/SVD_on_SAE_univ_v2.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"f13f5e667d8b9ac6"}},{"code_sha256_prefix":"046d60a7ed46bb56","entry":"compare_singular_values","repo":"wlg1/univ_feat_geom","repo_kind":"official","path":"_old_code/nbs_as_py/SVD_on_SAE_univ/SVD_on_SAE_univ_v3.py","file_url":"https://github.com/wlg1/univ_feat_geom/blob/HEAD/_old_code/nbs_as_py/SVD_on_SAE_univ/SVD_on_SAE_univ_v3.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"046d60a7ed46bb56"}},{"code_sha256_prefix":"8827ccff4f9a1386","entry":"generate_rand_svd","repo":"wlg1/univ_feat_geom","repo_kind":"official","path":"_old_code/nbs_as_py/SVD_on_SAE_univ/SVD_on_SAE_univ_v2.py","file_url":"https://github.com/wlg1/univ_feat_geom/blob/HEAD/_old_code/nbs_as_py/SVD_on_SAE_univ/SVD_on_SAE_univ_v2.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"8827ccff4f9a1386"}},{"code_sha256_prefix":"ec25fc71f56b3ad2","entry":"highest_correlation_rows","repo":"wlg1/univ_feat_geom","repo_kind":"official","path":"modal_scripts/fig_2_metrics.py","file_url":"https://github.com/wlg1/univ_feat_geom/blob/HEAD/modal_scripts/fig_2_metrics.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"ec25fc71f56b3ad2"}},{"code_sha256_prefix":"d6be561c21531bd0","entry":"normalize_singular_values","repo":"wlg1/univ_feat_geom","repo_kind":"official","path":"_old_code/nbs_as_py/SVD_on_SAE_univ/svd_on_SAE_univ_v1.py","file_url":"https://github.com/wlg1/univ_feat_geom/blob/HEAD/_old_code/nbs_as_py/SVD_on_SAE_univ/svd_on_SAE_univ_v1.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"d6be561c21531bd0"}},{"code_sha256_prefix":"a12456a671761d2c","entry":"pca_pandas","repo":"wlg1/univ_feat_geom","repo_kind":"official","path":"_old_code/nbs_as_py/feature_geom_explora/feature_weights_pca.py","file_url":"https://github.com/wlg1/univ_feat_geom/blob/HEAD/_old_code/nbs_as_py/feature_geom_explora/feature_weights_pca.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"a12456a671761d2c"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}