{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/q-s5-towards-quantized-state-space-models","title":"Q-S5: Towards Quantized State Space Models","arxiv_id":"2406.09477","date":"2024-06-13","proceeding":null,"authors":["Steven Abreu","Jens E. Pedersen","Kade M. Heckel","Alessandro Pierro"],"abstract":"In the quest for next-generation sequence modeling architectures, State Space Models (SSMs) have emerged as a potent alternative to transformers, particularly for their computational efficiency and suitability for dynamical systems. This paper investigates the effect of quantization on the S5 model to understand its impact on model performance and to facilitate its deployment to edge and resource-constrained platforms. Using quantization-aware training (QAT) and post-training quantization (PTQ), we systematically evaluate the quantization sensitivity of SSMs across different tasks like dynamical systems modeling, Sequential MNIST (sMNIST) and most of the Long Range Arena (LRA). We present fully quantized S5 models whose test accuracy drops less than 1% on sMNIST and most of the LRA. We find that performance on most tasks degrades significantly for recurrent weights below 8-bit precision, but that other components can be compressed further without significant loss of performance. Our results further show that PTQ only performs well on language-based LRA tasks whereas all others require QAT. Our investigation provides necessary insights for the continued development of efficient and hardware-optimized SSMs.","url_abs":"https://arxiv.org/abs/2406.09477v1","url_pdf":"https://arxiv.org/pdf/2406.09477v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"q-s5-towards-quantized-state-space-models","repo_url":"https://github.com/kmheckel/q-S5","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"jax","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"computational-efficiency","task_name":"Computational Efficiency"},{"task_slug":"quantization","task_name":"Quantization"},{"task_slug":"state-space-models","task_name":"State Space Models"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2406.09477","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2406.09477"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/kmheckel/q-S5","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":5,"unverified":1},"by_repo_kind":{"official":{"samples":6,"ran":5,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"1fe21a635f45224e","entry":"dysts_collate_fn","repo":"kmheckel/q-S5","repo_kind":"official","path":"experiments/dataloaders.py","file_url":"https://github.com/kmheckel/q-S5/blob/HEAD/experiments/dataloaders.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"1fe21a635f45224e"}},{"code_sha256_prefix":"d2ce6403f237ec60","entry":"dysts_data_loader","repo":"kmheckel/q-S5","repo_kind":"official","path":"experiments/dataloaders.py","file_url":"https://github.com/kmheckel/q-S5/blob/HEAD/experiments/dataloaders.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d2ce6403f237ec60"}},{"code_sha256_prefix":"eb453d02c991bcbb","entry":"int_or_none","repo":"kmheckel/q-S5","repo_kind":"official","path":"experiments/run_experiment.py","file_url":"https://github.com/kmheckel/q-S5/blob/HEAD/experiments/run_experiment.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"eb453d02c991bcbb"}},{"code_sha256_prefix":"9ea5b239c9627b3d","entry":"quant_dot_for_hadamard","repo":"kmheckel/q-S5","repo_kind":"official","path":"debugging/profiling/profile_aqt.py","file_url":"https://github.com/kmheckel/q-S5/blob/HEAD/debugging/profiling/profile_aqt.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"9ea5b239c9627b3d"}},{"code_sha256_prefix":"911f8a305d05a5f8","entry":"smape","repo":"kmheckel/q-S5","repo_kind":"official","path":"experiments/run_experiment.py","file_url":"https://github.com/kmheckel/q-S5/blob/HEAD/experiments/run_experiment.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"911f8a305d05a5f8"}},{"code_sha256_prefix":"19f57fed42a0bf7b","entry":"list_datasets","repo":"kmheckel/q-S5","repo_kind":"official","path":"experiments/dataloaders.py","file_url":"https://github.com/kmheckel/q-S5/blob/HEAD/experiments/dataloaders.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"19f57fed42a0bf7b"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}