{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/codebook-features-sparse-and-discrete","title":"Codebook Features: Sparse and Discrete Interpretability for Neural Networks","arxiv_id":"2310.17230","date":"2023-10-26","proceeding":null,"authors":["Alex Tamkin","Mohammad Taufeeque","Noah D. Goodman"],"abstract":"Understanding neural networks is challenging in part because of the dense, continuous nature of their hidden states. We explore whether we can train neural networks to have hidden states that are sparse, discrete, and more interpretable by quantizing their continuous features into what we call codebook features. Codebook features are produced by finetuning neural networks with vector quantization bottlenecks at each layer, producing a network whose hidden features are the sum of a small number of discrete vector codes chosen from a larger codebook. Surprisingly, we find that neural networks can operate under this extreme bottleneck with only modest degradation in performance. This sparse, discrete bottleneck also provides an intuitive way of controlling neural network behavior: first, find codes that activate when the desired behavior is present, then activate those same codes during generation to elicit that behavior. We validate our approach by training codebook Transformers on several different datasets. First, we explore a finite state machine dataset with far more hidden states than neurons. In this setting, our approach overcomes the superposition problem by assigning states to distinct codes, and we find that we can make the neural network behave as if it is in a different state by activating the code for that state. Second, we train Transformer language models with up to 410M parameters on two natural language datasets. We identify codes in these models representing diverse, disentangled concepts (ranging from negative emotions to months of the year) and find that we can guide the model to generate different topics by activating the appropriate codes during inference. Overall, codebook features appear to be a promising unit of analysis and control for neural networks and interpretability. Our codebase and models are open-sourced at https://github.com/taufeeque9/codebook-features.","url_abs":"https://arxiv.org/abs/2310.17230v1","url_pdf":"https://arxiv.org/pdf/2310.17230v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"codebook-features-sparse-and-discrete","repo_url":"https://github.com/taufeeque9/codebook-features","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"quantization","task_name":"Quantization"}],"methods":[{"method_slug":"absolute-position-encodings","method_name":"Absolute Position Encodings"},{"method_slug":"adam","method_name":"Adam"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"bpe","method_name":"BPE"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"label-smoothing","method_name":"Label Smoothing"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"position-wise-feed-forward-layer","method_name":"Position-Wise Feed-Forward Layer"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"transformer","method_name":"Transformer"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2310.17230","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2310.17230"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/taufeeque9/codebook-features","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":8,"ran_honours":1,"unverified":2},"by_repo_kind":{"official":{"samples":11,"ran":9,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"315c612a7005019a","entry":"color_str","repo":"taufeeque9/codebook-features","repo_kind":"official","path":"codebook_features/utils.py","file_url":"https://github.com/taufeeque9/codebook-features/blob/HEAD/codebook_features/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"315c612a7005019a"}},{"code_sha256_prefix":"e7a3a433b5807b51","entry":"create_tokenizer","repo":"taufeeque9/codebook-features","repo_kind":"official","path":"codebook_features/train_fsm_model.py","file_url":"https://github.com/taufeeque9/codebook-features/blob/HEAD/codebook_features/train_fsm_model.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e7a3a433b5807b51"}},{"code_sha256_prefix":"e883b9a34464b3db","entry":"features_to_tokens","repo":"taufeeque9/codebook-features","repo_kind":"official","path":"codebook_features/utils.py","file_url":"https://github.com/taufeeque9/codebook-features/blob/HEAD/codebook_features/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e883b9a34464b3db"}},{"code_sha256_prefix":"bc3765261beb68f8","entry":"get_next_state_probs","repo":"taufeeque9/codebook-features","repo_kind":"official","path":"codebook_features/fsm_utils.py","file_url":"https://github.com/taufeeque9/codebook-features/blob/HEAD/codebook_features/fsm_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"bc3765261beb68f8"}},{"code_sha256_prefix":"ca5c4697a5c093ec","entry":"get_optimizer","repo":"taufeeque9/codebook-features","repo_kind":"official","path":"codebook_features/train_codebook.py","file_url":"https://github.com/taufeeque9/codebook-features/blob/HEAD/codebook_features/train_codebook.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ca5c4697a5c093ec"}},{"code_sha256_prefix":"0baf689e42d326b6","entry":"load_code_search_cache","repo":"taufeeque9/codebook-features","repo_kind":"official","path":"codebook_features/code_search_utils.py","file_url":"https://github.com/taufeeque9/codebook-features/blob/HEAD/codebook_features/code_search_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"0baf689e42d326b6"}},{"code_sha256_prefix":"dbc3ad84e8a2ddd9","entry":"prepare_logging","repo":"taufeeque9/codebook-features","repo_kind":"official","path":"codebook_features/train_codebook.py","file_url":"https://github.com/taufeeque9/codebook-features/blob/HEAD/codebook_features/train_codebook.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"dbc3ad84e8a2ddd9"}},{"code_sha256_prefix":"d2895bdf14e8f94f","entry":"preprocess_logits_for_metrics","repo":"taufeeque9/codebook-features","repo_kind":"official","path":"codebook_features/run_clm.py","file_url":"https://github.com/taufeeque9/codebook-features/blob/HEAD/codebook_features/run_clm.py","link_basis":"harvester_set","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d2895bdf14e8f94f"}},{"code_sha256_prefix":"605b9653cd77b330","entry":"search_re","repo":"taufeeque9/codebook-features","repo_kind":"official","path":"codebook_features/code_search_utils.py","file_url":"https://github.com/taufeeque9/codebook-features/blob/HEAD/codebook_features/code_search_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"605b9653cd77b330"}},{"code_sha256_prefix":"4471d92715bb6e40","entry":"load_dataset_cache","repo":"taufeeque9/codebook-features","repo_kind":"official","path":"codebook_features/code_search_utils.py","file_url":"https://github.com/taufeeque9/codebook-features/blob/HEAD/codebook_features/code_search_utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"4471d92715bb6e40"}},{"code_sha256_prefix":"1c6d09ac6435d64a","entry":"logits_to_pred","repo":"taufeeque9/codebook-features","repo_kind":"official","path":"codebook_features/utils.py","file_url":"https://github.com/taufeeque9/codebook-features/blob/HEAD/codebook_features/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"1c6d09ac6435d64a"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}