{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/analyzing-in-abilities-of-saes-via-formal","title":"Analyzing (In)Abilities of SAEs via Formal Languages","arxiv_id":"2410.11767","date":"2024-10-15","proceeding":null,"authors":["Abhinav Menon","Manish Shrivastava","David Krueger","Ekdeep Singh Lubana"],"abstract":"Autoencoders have been used for finding interpretable and disentangled features underlying neural network representations in both image and text domains. While the efficacy and pitfalls of such methods are well-studied in vision, there is a lack of corresponding results, both qualitative and quantitative, for the text domain. We aim to address this gap by training sparse autoencoders (SAEs) on a synthetic testbed of formal languages. Specifically, we train SAEs on the hidden representations of models trained on formal languages (Dyck-2, Expr, and English PCFG) under a wide variety of hyperparameter settings, finding interpretable latents often emerge in the features learned by our SAEs. However, similar to vision, we find performance turns out to be highly sensitive to inductive biases of the training pipeline. Moreover, we show latents correlating to certain features of the input do not always induce a causal impact on model's computation. We thus argue that causality has to become a central target in SAE training: learning of causal features should be incentivized from the ground-up. Motivated by this, we propose and perform preliminary investigations for an approach that promotes learning of causally relevant features in our formal language setting.","url_abs":"https://arxiv.org/abs/2410.11767v3","url_pdf":"https://arxiv.org/pdf/2410.11767v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2410.11767","atlas_url":"https://app.syntology.ai/?focus=2410.11767","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2410.11767"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/Abhinav271828/pcfg-sae-causa","reach":{"status":"gone","observed_at":"2026-09-17","how":"tree_404+repo_404"}},{"provenance":"deterministic:regex_extraction","url":"https://github.com/Abhinav271828/pcfg-sae-causal-arr-oct24","reach":{"status":"ok"}},{"provenance":"deterministic:regex_extraction","url":"https://github.com/Abhinav271828/pcfg-sae-cau","reach":{"status":"gone","observed_at":"2026-09-17","how":"tree_404+repo_404"}}],"summary":{"ran":3,"unverified":7},"by_repo_kind":{"found_in_text":{"samples":10,"ran":3,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":10,"samples":[{"code_sha256_prefix":"8f5c38520a624a74","entry":"dec2base","repo":"Abhinav271828/pcfg-sae-causal-arr-oct24","repo_kind":"found_in_text","path":"datasets/utils.py","file_url":"https://github.com/Abhinav271828/pcfg-sae-causal-arr-oct24/blob/HEAD/datasets/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"8f5c38520a624a74"}},{"code_sha256_prefix":"4138f619ef3ab02c","entry":"define_prior","repo":"Abhinav271828/pcfg-sae-causal-arr-oct24","repo_kind":"found_in_text","path":"dgp/utils.py","file_url":"https://github.com/Abhinav271828/pcfg-sae-causal-arr-oct24/blob/HEAD/dgp/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"4138f619ef3ab02c"}},{"code_sha256_prefix":"bfa0b6ce30af730f","entry":"get_module","repo":"Abhinav271828/pcfg-sae-causal-arr-oct24","repo_kind":"found_in_text","path":"utils/analysis.py","file_url":"https://github.com/Abhinav271828/pcfg-sae-causal-arr-oct24/blob/HEAD/utils/analysis.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"bfa0b6ce30af730f"}},{"code_sha256_prefix":"9bf1df20447a502c","entry":"dec2bin","repo":"Abhinav271828/pcfg-sae-causal-arr-oct24","repo_kind":"found_in_text","path":"datasets/utils.py","file_url":"https://github.com/Abhinav271828/pcfg-sae-causal-arr-oct24/blob/HEAD/datasets/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"9bf1df20447a502c"}},{"code_sha256_prefix":"834a92ff49752de4","entry":"get_dataloader","repo":"Abhinav271828/pcfg-sae-causal-arr-oct24","repo_kind":"found_in_text","path":"dgp/dataloader.py","file_url":"https://github.com/Abhinav271828/pcfg-sae-causal-arr-oct24/blob/HEAD/dgp/dataloader.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"834a92ff49752de4"}},{"code_sha256_prefix":"4b85b94877cbb65a","entry":"grammar_evals","repo":"Abhinav271828/pcfg-sae-causal-arr-oct24","repo_kind":"found_in_text","path":"evals/evals.py","file_url":"https://github.com/Abhinav271828/pcfg-sae-causal-arr-oct24/blob/HEAD/evals/evals.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"4b85b94877cbb65a"}},{"code_sha256_prefix":"50cccbfce6dd484d","entry":"sample_data_from_generator_classes","repo":"Abhinav271828/pcfg-sae-causal-arr-oct24","repo_kind":"found_in_text","path":"datasets/random_hierarchy_model.py","file_url":"https://github.com/Abhinav271828/pcfg-sae-causal-arr-oct24/blob/HEAD/datasets/random_hierarchy_model.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"50cccbfce6dd484d"}},{"code_sha256_prefix":"acc3479a59a6c04c","entry":"sample_rules","repo":"Abhinav271828/pcfg-sae-causal-arr-oct24","repo_kind":"found_in_text","path":"datasets/random_hierarchy_model.py","file_url":"https://github.com/Abhinav271828/pcfg-sae-causal-arr-oct24/blob/HEAD/datasets/random_hierarchy_model.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"acc3479a59a6c04c"}},{"code_sha256_prefix":"779207a23dcaeecb","entry":"sample_with_replacement","repo":"Abhinav271828/pcfg-sae-causal-arr-oct24","repo_kind":"found_in_text","path":"datasets/random_hierarchy_model.py","file_url":"https://github.com/Abhinav271828/pcfg-sae-causal-arr-oct24/blob/HEAD/datasets/random_hierarchy_model.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"779207a23dcaeecb"}},{"code_sha256_prefix":"4088666d0047518a","entry":"top_k_top_p_filtering","repo":"Abhinav271828/pcfg-sae-causal-arr-oct24","repo_kind":"found_in_text","path":"model/model.py","file_url":"https://github.com/Abhinav271828/pcfg-sae-causal-arr-oct24/blob/HEAD/model/model.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"4088666d0047518a"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}