{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/it-s-raw-audio-generation-with-state-space","title":"It's Raw! Audio Generation with State-Space Models","arxiv_id":"2202.09729","date":"2022-02-20","proceeding":null,"authors":["Karan Goel","Albert Gu","Chris Donahue","Christopher Ré"],"abstract":"Developing architectures suitable for modeling raw audio is a challenging problem due to the high sampling rates of audio waveforms. Standard sequence modeling approaches like RNNs and CNNs have previously been tailored to fit the demands of audio, but the resultant architectures make undesirable computational tradeoffs and struggle to model waveforms effectively. We propose SaShiMi, a new multi-scale architecture for waveform modeling built around the recently introduced S4 model for long sequence modeling. We identify that S4 can be unstable during autoregressive generation, and provide a simple improvement to its parameterization by drawing connections to Hurwitz matrices. SaShiMi yields state-of-the-art performance for unconditional waveform generation in the autoregressive setting. Additionally, SaShiMi improves non-autoregressive generation performance when used as the backbone architecture for a diffusion model. Compared to prior architectures in the autoregressive generation setting, SaShiMi generates piano and speech waveforms which humans find more musical and coherent respectively, e.g. 2x better mean opinion scores than WaveNet on an unconditional speech generation task. On a music generation task, SaShiMi outperforms WaveNet on density estimation and speed at both training and inference even when using 3x fewer parameters. Code can be found at https://github.com/HazyResearch/state-spaces and samples at https://hazyresearch.stanford.edu/sashimi-examples.","url_abs":"https://arxiv.org/abs/2202.09729v1","url_pdf":"https://arxiv.org/pdf/2202.09729v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"it-s-raw-audio-generation-with-state-space","repo_url":"https://github.com/hazyresearch/state-spaces","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}},{"paper_slug":"it-s-raw-audio-generation-with-state-space","repo_url":"https://github.com/ag1988/dss","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"it-s-raw-audio-generation-with-state-space","repo_url":"https://github.com/albertfgu/diffwave-sashimi","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"it-s-raw-audio-generation-with-state-space","repo_url":"https://github.com/necrashter/SaShiMi-796","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"it-s-raw-audio-generation-with-state-space","repo_url":"https://github.com/rf5/diffwave-unconditional","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"it-s-raw-audio-generation-with-state-space","repo_url":"https://github.com/rf5/simple-sashimi","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"audio-generation","task_name":"Audio Generation"},{"task_slug":"density-estimation","task_name":"Density Estimation"},{"task_slug":"music-generation","task_name":"Music Generation"},{"task_slug":"state-space-models","task_name":"State Space Models"}],"methods":[{"method_slug":"diffusion","method_name":"Diffusion"},{"method_slug":"dilated-causal-convolution","method_name":"Dilated Causal Convolution"},{"method_slug":"mixture-of-logistic-distributions","method_name":"Mixture of Logistic Distributions"},{"method_slug":"speed","method_name":"SPEED"},{"method_slug":"wavenet","method_name":"WaveNet"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2202.09729","atlas_url":"https://app.syntology.ai/?focus=2202.09729","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2202.09729"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/ag1988/dss","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/rf5/simple-sashimi","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/albertfgu/diffwave-sashimi","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/necrashter/SaShiMi-796","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/hazyresearch/state-spaces","reach":{"status":"ok","spdx":"Apache-2.0"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/rf5/diffwave-unconditional","reach":null}],"summary":{"ran_draft_wrong":4,"ran_fixture":2},"by_repo_kind":{"listed":{"samples":3,"ran":3,"repositories":3}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":4,"samples":[{"code_sha256_prefix":"0f786c407fb1ee4c","entry":"swish","repo":null,"repo_kind":null,"path":null,"file_url":null,"link_basis":"identical_code_first_harvested_elsewhere","language":"python","status":"ran_draft_wrong","verification_level":2,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":null,"inline_ok":false,"mcp_get_code":{"code_sha256":"0f786c407fb1ee4c"}},{"code_sha256_prefix":"89b10a8b26bbb7ed","entry":"Activation","repo":null,"repo_kind":null,"path":null,"file_url":null,"link_basis":"identical_code_first_harvested_elsewhere","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":null,"inline_ok":false,"mcp_get_code":{"code_sha256":"89b10a8b26bbb7ed"}},{"code_sha256_prefix":"163666f807c3a9bd","entry":"calc_diffusion_step_embedding","repo":"rf5/simple-sashimi","repo_kind":"listed","path":"sashimi/model.py","file_url":"https://github.com/rf5/simple-sashimi/blob/HEAD/sashimi/model.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"163666f807c3a9bd"}},{"code_sha256_prefix":"f3449af8fae1d130","entry":"get_initializer","repo":null,"repo_kind":null,"path":null,"file_url":null,"link_basis":"identical_code_first_harvested_elsewhere","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":null,"inline_ok":false,"mcp_get_code":{"code_sha256":"f3449af8fae1d130"}},{"code_sha256_prefix":"5aa881c74c73bcb8","entry":"naive_cauchy_kernel","repo":"necrashter/SaShiMi-796","repo_kind":"listed","path":"S4/cauchy.py","file_url":"https://github.com/necrashter/SaShiMi-796/blob/HEAD/S4/cauchy.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"5aa881c74c73bcb8"}},{"code_sha256_prefix":"8765bfcc420abc33","entry":"sampling","repo":"albertfgu/diffwave-sashimi","repo_kind":"listed","path":"generate.py","file_url":"https://github.com/albertfgu/diffwave-sashimi/blob/HEAD/generate.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"8765bfcc420abc33"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}