{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/stochastic-parameter-decomposition","title":"Stochastic Parameter Decomposition","arxiv_id":"2506.20790","date":"2025-06-25","proceeding":null,"authors":["Lucius Bushnaq","Dan Braun","Lee Sharkey"],"abstract":"A key step in reverse engineering neural networks is to decompose them into simpler parts that can be studied in relative isolation. Linear parameter decomposition -- a framework that has been proposed to resolve several issues with current decomposition methods -- decomposes neural network parameters into a sum of sparsely used vectors in parameter space. However, the current main method in this framework, Attribution-based Parameter Decomposition (APD), is impractical on account of its computational cost and sensitivity to hyperparameters. In this work, we introduce \\textit{Stochastic Parameter Decomposition} (SPD), a method that is more scalable and robust to hyperparameters than APD, which we demonstrate by decomposing models that are slightly larger and more complex than was possible to decompose with APD. We also show that SPD avoids other issues, such as shrinkage of the learned parameters, and better identifies ground truth mechanisms in toy models. By bridging causal mediation analysis and network decomposition methods, this demonstration opens up new research possibilities in mechanistic interpretability by removing barriers to scaling linear parameter decomposition methods to larger models. We release a library for running SPD and reproducing our experiments at https://github.com/goodfire-ai/spd.","url_abs":"https://arxiv.org/abs/2506.20790v1","url_pdf":"https://arxiv.org/pdf/2506.20790v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"stochastic-parameter-decomposition","repo_url":"https://github.com/goodfire-ai/spd","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"jax","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2506.20790","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2506.20790"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/goodfire-ai/spd","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"unverified":8},"by_repo_kind":{"official":{"samples":8,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"b4895ccdc0ffa853","entry":"flatten_typed_lists","repo":"goodfire-ai/spd","repo_kind":"official","path":"param_decomp/core/configs.py","file_url":"https://github.com/goodfire-ai/spd/blob/HEAD/param_decomp/core/configs.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b4895ccdc0ffa853"}},{"code_sha256_prefix":"b3d764023ba424a2","entry":"lower_leaky_hard_sigmoid","repo":"goodfire-ai/spd","repo_kind":"official","path":"param_decomp/core/ci_fn.py","file_url":"https://github.com/goodfire-ai/spd/blob/HEAD/param_decomp/core/ci_fn.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b3d764023ba424a2"}},{"code_sha256_prefix":"d642a165da5892e9","entry":"make_read_only_checkpoint_manager","repo":"goodfire-ai/spd","repo_kind":"official","path":"param_decomp/core/checkpoint.py","file_url":"https://github.com/goodfire-ai/spd/blob/HEAD/param_decomp/core/checkpoint.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d642a165da5892e9"}},{"code_sha256_prefix":"9cb9a457a77e272d","entry":"map_site_ci","repo":"goodfire-ai/spd","repo_kind":"official","path":"param_decomp/core/components.py","file_url":"https://github.com/goodfire-ai/spd/blob/HEAD/param_decomp/core/components.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"9cb9a457a77e272d"}},{"code_sha256_prefix":"4b527659babe5283","entry":"resolve_site_groups","repo":"goodfire-ai/spd","repo_kind":"official","path":"param_decomp/core/ci_l0_eval.py","file_url":"https://github.com/goodfire-ai/spd/blob/HEAD/param_decomp/core/ci_l0_eval.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"4b527659babe5283"}},{"code_sha256_prefix":"b6cb8c88559d2649","entry":"site_ci_values","repo":"goodfire-ai/spd","repo_kind":"official","path":"param_decomp/core/components.py","file_url":"https://github.com/goodfire-ai/spd/blob/HEAD/param_decomp/core/components.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b6cb8c88559d2649"}},{"code_sha256_prefix":"01b5e9436908ad0b","entry":"store_unit_float","repo":"goodfire-ai/spd","repo_kind":"official","path":"param_decomp/core/adversary.py","file_url":"https://github.com/goodfire-ai/spd/blob/HEAD/param_decomp/core/adversary.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"01b5e9436908ad0b"}},{"code_sha256_prefix":"887f54083073694e","entry":"upper_leaky_hard_sigmoid","repo":"goodfire-ai/spd","repo_kind":"official","path":"param_decomp/core/ci_fn.py","file_url":"https://github.com/goodfire-ai/spd/blob/HEAD/param_decomp/core/ci_fn.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"887f54083073694e"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}