{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/arxiv-2602-04718","title":"Superposition Without Interference? Towards Isolated Interventions via Almost Orthogonal Features in Language Models","arxiv_id":"2602.04718","date":"2026-02-04","proceeding":null,"authors":["Moritz Miller","Florent Draye","Bernhard Schölkopf"],"abstract":"A central premise in mechanistic interpretability is that meaningful concepts in language models are represented by linear features in activation space. For such features to support reliable interventions, manipulating one feature should not substantially alter the effects of others. In practice, however, feature entanglement leads to interference such that localized interventions can have unintended downstream effects. Motivated by the _Independent Causal Mechanisms_ principle, we propose to constrain internal features to be almost orthogonal. We argue that this promotes modular representations amenable to causal intervention. We formalize this problem by characterizing the gap between an idealized isolated intervention and its realized effect on model outputs in terms of feature interference. We upper-bound the propagation of feature interference in terms of the self-coherence of the feature dictionary, and relate this discrepancy to an explicit orthogonality regularization on the dictionary itself. Empirically, we show that this regularization enables more isolated interventions on mathematical reasoning concepts while preserving model performance. Our code is available under https://github.com/mrtzmllr/sae-icm.","url_abs":"https://arxiv.org/abs/2602.04718","url_pdf":"https://arxiv.org/pdf/2602.04718","source":{"archive":null,"snapshot":"2025-07-28","note":"not in the Papers with Code archive (frozen at the snapshot)","row_kind":"graph","title_abstract_authors_date":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)"},"code_links":[],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2602.04718","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2602.04718"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"mentioned_in_github":null,"is_official":null,"provenance":"deterministic:regex_extraction","mentioned_in_paper":null,"url":"https://github.com/mrtzmllr/sae-icm","reach":null}],"summary":{"ran_draft_wrong":6,"unverified":5},"by_repo_kind":{"found_in_text":{"samples":11,"ran":6,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":11,"samples":[{"code_sha256_prefix":"7029189e239ecbba","entry":"_include_weights_in_pathname","repo":"mrtzmllr/sae-icm","repo_kind":"found_in_text","path":"src/poet/orthogonality.py","file_url":"https://github.com/mrtzmllr/sae-icm/blob/HEAD/src/poet/orthogonality.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"7029189e239ecbba"}},{"code_sha256_prefix":"5919ca460891911e","entry":"_initial_parts","repo":"mrtzmllr/sae-icm","repo_kind":"found_in_text","path":"src/poet/orthogonality.py","file_url":"https://github.com/mrtzmllr/sae-icm/blob/HEAD/src/poet/orthogonality.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"5919ca460891911e"}},{"code_sha256_prefix":"d4b4a13a474772dd","entry":"_return_tril","repo":"mrtzmllr/sae-icm","repo_kind":"found_in_text","path":"src/poet/orthogonality.py","file_url":"https://github.com/mrtzmllr/sae-icm/blob/HEAD/src/poet/orthogonality.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"d4b4a13a474772dd"}},{"code_sha256_prefix":"1783c3f987a62543","entry":"insertion_layer_tag","repo":"mrtzmllr/sae-icm","repo_kind":"found_in_text","path":"src/poet/orthogonality.py","file_url":"https://github.com/mrtzmllr/sae-icm/blob/HEAD/src/poet/orthogonality.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"1783c3f987a62543"}},{"code_sha256_prefix":"7118a3af4f3be8ea","entry":"write_feature_storage_dir","repo":"mrtzmllr/sae-icm","repo_kind":"found_in_text","path":"src/poet/orthogonality.py","file_url":"https://github.com/mrtzmllr/sae-icm/blob/HEAD/src/poet/orthogonality.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"7118a3af4f3be8ea"}},{"code_sha256_prefix":"42e67d9c61331b89","entry":"write_output_dir","repo":"mrtzmllr/sae-icm","repo_kind":"found_in_text","path":"src/poet/orthogonality.py","file_url":"https://github.com/mrtzmllr/sae-icm/blob/HEAD/src/poet/orthogonality.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"42e67d9c61331b89"}},{"code_sha256_prefix":"fd16bc0d7bcf6d38","entry":"_active_features_decoder","repo":"mrtzmllr/sae-icm","repo_kind":"found_in_text","path":"src/poet/orthogonality.py","file_url":"https://github.com/mrtzmllr/sae-icm/blob/HEAD/src/poet/orthogonality.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"fd16bc0d7bcf6d38"}},{"code_sha256_prefix":"902f1860466d8df1","entry":"_orthogonality_loss","repo":"mrtzmllr/sae-icm","repo_kind":"found_in_text","path":"src/poet/orthogonality.py","file_url":"https://github.com/mrtzmllr/sae-icm/blob/HEAD/src/poet/orthogonality.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"902f1860466d8df1"}},{"code_sha256_prefix":"0579c9c094b662cc","entry":"_retrieve_decoder_weight","repo":"mrtzmllr/sae-icm","repo_kind":"found_in_text","path":"src/poet/orthogonality.py","file_url":"https://github.com/mrtzmllr/sae-icm/blob/HEAD/src/poet/orthogonality.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"0579c9c094b662cc"}},{"code_sha256_prefix":"db189d0fa3e9c6a6","entry":"_sampled_orthogonal_features","repo":"mrtzmllr/sae-icm","repo_kind":"found_in_text","path":"src/poet/orthogonality.py","file_url":"https://github.com/mrtzmllr/sae-icm/blob/HEAD/src/poet/orthogonality.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"db189d0fa3e9c6a6"}},{"code_sha256_prefix":"be04c29cea958961","entry":"retrieve_feature_dict","repo":"mrtzmllr/sae-icm","repo_kind":"found_in_text","path":"src/poet/orthogonality.py","file_url":"https://github.com/mrtzmllr/sae-icm/blob/HEAD/src/poet/orthogonality.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"be04c29cea958961"}}]},"arxiv_metadata":{"licence":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)","fields":["title","abstract","authors","date"],"primary_category":"cs.LG","source":"arxiv_2026.jsonl"},"syntology_extracted_results":null}