{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/discrete-key-value-bottleneck","title":"Discrete Key-Value Bottleneck","arxiv_id":"2207.11240","date":"2022-07-22","proceeding":null,"authors":["Frederik Träuble","Anirudh Goyal","Nasim Rahaman","Michael Mozer","Kenji Kawaguchi","Yoshua Bengio","Bernhard Schölkopf"],"abstract":"Deep neural networks perform well on classification tasks where data streams are i.i.d. and labeled data is abundant. Challenges emerge with non-stationary training data streams such as continual learning. One powerful approach that has addressed this challenge involves pre-training of large encoders on volumes of readily available data, followed by task-specific tuning. Given a new task, however, updating the weights of these encoders is challenging as a large number of weights needs to be fine-tuned, and as a result, they forget information about the previous tasks. In the present work, we propose a model architecture to address this issue, building upon a discrete bottleneck containing pairs of separate and learnable key-value codes. Our paradigm will be to encode; process the representation via a discrete bottleneck; and decode. Here, the input is fed to the pre-trained encoder, the output of the encoder is used to select the nearest keys, and the corresponding values are fed to the decoder to solve the current task. The model can only fetch and re-use a sparse number of these key-value pairs during inference, enabling localized and context-dependent model updates. We theoretically investigate the ability of the discrete key-value bottleneck to minimize the effect of learning under distribution shifts and show that it reduces the complexity of the hypothesis class. We empirically verify the proposed method under challenging class-incremental learning scenarios and show that the proposed model - without any task boundaries - reduces catastrophic forgetting across a wide variety of pre-trained models, outperforming relevant baselines on this task.","url_abs":"https://arxiv.org/abs/2207.11240v3","url_pdf":"https://arxiv.org/pdf/2207.11240v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"discrete-key-value-bottleneck","repo_url":"https://github.com/ftraeuble/experiments_discrete_key_value_bottleneck","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"class-incremental-learning","task_name":"Class Incremental Learning"},{"task_slug":"continual-learning","task_name":"Continual Learning"},{"task_slug":"incremental-learning","task_name":"Incremental Learning"},{"task_slug":"class-incremental-learning-1","task_name":"class-incremental learning"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2207.11240","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2207.11240"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/ftraeuble/experiments_discrete_key_value_bottleneck","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran_violates":2,"unverified":7},"by_repo_kind":{"official":{"samples":7,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":2,"samples":[{"code_sha256_prefix":"60fff7c3c400d7ff","entry":"default","repo":null,"repo_kind":null,"path":null,"file_url":null,"link_basis":"identical_code_first_harvested_elsewhere","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"well_formed","behaviour_fingerprint":true,"licence":null,"inline_ok":false,"mcp_get_code":{"code_sha256":"60fff7c3c400d7ff"}},{"code_sha256_prefix":"aa5486a3650902d8","entry":"exists","repo":null,"repo_kind":null,"path":null,"file_url":null,"link_basis":"identical_code_first_harvested_elsewhere","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":null,"inline_ok":false,"mcp_get_code":{"code_sha256":"aa5486a3650902d8"}},{"code_sha256_prefix":"746e0f9d6857f472","entry":"embedding_dataset_exists","repo":"ftraeuble/experiments_discrete_key_value_bottleneck","repo_kind":"official","path":"kv_bottleneck_experiments/utils/data.py","file_url":"https://github.com/ftraeuble/experiments_discrete_key_value_bottleneck/blob/HEAD/kv_bottleneck_experiments/utils/data.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"746e0f9d6857f472"}},{"code_sha256_prefix":"e9304c4dd4be4d62","entry":"get_class_nums","repo":"ftraeuble/experiments_discrete_key_value_bottleneck","repo_kind":"official","path":"kv_bottleneck_experiments/utils/base.py","file_url":"https://github.com/ftraeuble/experiments_discrete_key_value_bottleneck/blob/HEAD/kv_bottleneck_experiments/utils/base.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e9304c4dd4be4d62"}},{"code_sha256_prefix":"05b24aa01f9569e1","entry":"get_embedding_dataloader","repo":"ftraeuble/experiments_discrete_key_value_bottleneck","repo_kind":"official","path":"kv_bottleneck_experiments/utils/data.py","file_url":"https://github.com/ftraeuble/experiments_discrete_key_value_bottleneck/blob/HEAD/kv_bottleneck_experiments/utils/data.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"05b24aa01f9569e1"}},{"code_sha256_prefix":"de459db4d1d897e2","entry":"get_encoder_output_shape","repo":"ftraeuble/experiments_discrete_key_value_bottleneck","repo_kind":"official","path":"kv_bottleneck_experiments/utils/model.py","file_url":"https://github.com/ftraeuble/experiments_discrete_key_value_bottleneck/blob/HEAD/kv_bottleneck_experiments/utils/model.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"de459db4d1d897e2"}},{"code_sha256_prefix":"1e4efca94cf1e079","entry":"get_image_size","repo":"ftraeuble/experiments_discrete_key_value_bottleneck","repo_kind":"official","path":"kv_bottleneck_experiments/utils/base.py","file_url":"https://github.com/ftraeuble/experiments_discrete_key_value_bottleneck/blob/HEAD/kv_bottleneck_experiments/utils/base.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"1e4efca94cf1e079"}},{"code_sha256_prefix":"e9809af7e92e8792","entry":"get_resnet50_output_shape","repo":"ftraeuble/experiments_discrete_key_value_bottleneck","repo_kind":"official","path":"kv_bottleneck_experiments/utils/model.py","file_url":"https://github.com/ftraeuble/experiments_discrete_key_value_bottleneck/blob/HEAD/kv_bottleneck_experiments/utils/model.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e9809af7e92e8792"}},{"code_sha256_prefix":"c1432af58cb5e45a","entry":"get_split_dataloaders","repo":"ftraeuble/experiments_discrete_key_value_bottleneck","repo_kind":"official","path":"kv_bottleneck_experiments/utils/data.py","file_url":"https://github.com/ftraeuble/experiments_discrete_key_value_bottleneck/blob/HEAD/kv_bottleneck_experiments/utils/data.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c1432af58cb5e45a"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}