{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/hungry-hungry-hippos-towards-language","title":"Hungry Hungry Hippos: Towards Language Modeling with State Space Models","arxiv_id":"2212.14052","date":"2022-12-28","proceeding":null,"authors":["Daniel Y. Fu","Tri Dao","Khaled K. Saab","Armin W. Thomas","Atri Rudra","Christopher Ré"],"abstract":"State space models (SSMs) have demonstrated state-of-the-art sequence modeling performance in some modalities, but underperform attention in language modeling. Moreover, despite scaling nearly linearly in sequence length instead of quadratically, SSMs are still slower than Transformers due to poor hardware utilization. In this paper, we make progress on understanding the expressivity gap between SSMs and attention in language modeling, and on reducing the hardware barrier between SSMs and attention. First, we use synthetic language modeling tasks to understand the gap between SSMs and attention. We find that existing SSMs struggle with two capabilities: recalling earlier tokens in the sequence and comparing tokens across the sequence. To understand the impact on language modeling, we propose a new SSM layer, H3, that is explicitly designed for these abilities. H3 matches attention on the synthetic languages and comes within 0.4 PPL of Transformers on OpenWebText. Furthermore, a hybrid 125M-parameter H3-attention model that retains two attention layers surprisingly outperforms Transformers on OpenWebText by 1.0 PPL. Next, to improve the efficiency of training SSMs on modern hardware, we propose FlashConv. FlashConv uses a fused block FFT algorithm to improve efficiency on sequences up to 8K, and introduces a novel state passing algorithm that exploits the recurrent properties of SSMs to scale to longer sequences. FlashConv yields 2$\\times$ speedup on the long-range arena benchmark and allows hybrid language models to generate text 2.4$\\times$ faster than Transformers. Using FlashConv, we scale hybrid H3-attention language models up to 2.7B parameters on the Pile and find promising initial results, achieving lower perplexity than Transformers and outperforming Transformers in zero- and few-shot learning on a majority of tasks in the SuperGLUE benchmark.","url_abs":"https://arxiv.org/abs/2212.14052v3","url_pdf":"https://arxiv.org/pdf/2212.14052v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"hungry-hungry-hippos-towards-language","repo_url":"https://github.com/hazyresearch/h3","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}},{"paper_slug":"hungry-hungry-hippos-towards-language","repo_url":"https://github.com/hazyresearch/safari","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}},{"paper_slug":"hungry-hungry-hippos-towards-language","repo_url":"https://github.com/lindermanlab/S5","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"jax","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":null,"task_name":"8k"},{"task_slug":"coreference-resolution","task_name":"Coreference Resolution"},{"task_slug":"few-shot-learning","task_name":"Few-Shot Learning"},{"task_slug":"language-modeling","task_name":"Language Modeling"},{"task_slug":"language-modelling","task_name":"Language Modelling"},{"task_slug":"long-range-modeling","task_name":"Long-range modeling"},{"task_slug":"natural-language-inference","task_name":"Natural Language Inference"},{"task_slug":"question-answering","task_name":"Question Answering"},{"task_slug":"state-space-models","task_name":"State Space Models"},{"task_slug":"word-sense-disambiguation","task_name":"Word Sense Disambiguation"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/coreference-resolution-on-winograd-schema","task":"Coreference Resolution","dataset":"Winograd Schema Challenge","model":"H3 125M (3-shot, rank classification)","rank_in_archive_order":46,"of":82,"metrics":{"Accuracy":"63.5"},"uses_additional_data":false},{"leaderboard":"/sota/coreference-resolution-on-winograd-schema","task":"Coreference Resolution","dataset":"Winograd Schema Challenge","model":"H3 125M (0-shot, rank classification)","rank_in_archive_order":55,"of":82,"metrics":{"Accuracy":"61.5"},"uses_additional_data":false},{"leaderboard":"/sota/coreference-resolution-on-winograd-schema","task":"Coreference Resolution","dataset":"Winograd Schema Challenge","model":"Hybrid H3 125M (3-shot, logit scoring)","rank_in_archive_order":78,"of":82,"metrics":{"Accuracy":"43.3"},"uses_additional_data":false},{"leaderboard":"/sota/language-modelling-on-the-pile","task":"Language Modelling","dataset":"The Pile","model":"Hybrid H3 125M","rank_in_archive_order":29,"of":39,"metrics":{"Test perplexity":"10.2"},"uses_additional_data":false},{"leaderboard":"/sota/language-modelling-on-the-pile","task":"Language Modelling","dataset":"The Pile","model":"Transformer 125M","rank_in_archive_order":31,"of":39,"metrics":{"Test perplexity":"10.7"},"uses_additional_data":false},{"leaderboard":"/sota/language-modelling-on-wikitext-103","task":"Language Modelling","dataset":"WikiText-103","model":"Hybrid H3 (2.7B)","rank_in_archive_order":2,"of":89,"metrics":{"Number of params":"2700M","Test perplexity":"10.6"},"uses_additional_data":true},{"leaderboard":"/sota/language-modelling-on-wikitext-103","task":"Language Modelling","dataset":"WikiText-103","model":"Hybrid H3 (1.3B)","rank_in_archive_order":6,"of":89,"metrics":{"Number of params":"1300M","Test perplexity":"12.5"},"uses_additional_data":true},{"leaderboard":"/sota/language-modelling-on-wikitext-103","task":"Language Modelling","dataset":"WikiText-103","model":"Hybrid H3 (355M)","rank_in_archive_order":18,"of":89,"metrics":{"Number of params":"355M","Test perplexity":"16.9"},"uses_additional_data":true},{"leaderboard":"/sota/language-modelling-on-wikitext-103","task":"Language Modelling","dataset":"WikiText-103","model":"Hybrid H3 125M","rank_in_archive_order":38,"of":89,"metrics":{"Test perplexity":"18.5"},"uses_additional_data":false},{"leaderboard":"/sota/language-modelling-on-wikitext-103","task":"Language Modelling","dataset":"WikiText-103","model":"Hybrid H3 (125M)","rank_in_archive_order":53,"of":89,"metrics":{"Number of params":"125M","Test perplexity":"23.7"},"uses_additional_data":true},{"leaderboard":"/sota/natural-language-inference-on-rte","task":"Natural Language Inference","dataset":"RTE","model":"Hybrid H3 125M (0-shot, logit scoring)","rank_in_archive_order":74,"of":90,"metrics":{"Accuracy":"59.2%"},"uses_additional_data":false},{"leaderboard":"/sota/natural-language-inference-on-rte","task":"Natural Language Inference","dataset":"RTE","model":"Hybrid H3 125M (3-shot, logit scoring)","rank_in_archive_order":77,"of":90,"metrics":{"Accuracy":"58.1%"},"uses_additional_data":false},{"leaderboard":"/sota/natural-language-inference-on-rte","task":"Natural Language Inference","dataset":"RTE","model":"Hybrid H3 125M (3-shot, rank classification)","rank_in_archive_order":78,"of":90,"metrics":{"Accuracy":"58.1%"},"uses_additional_data":false},{"leaderboard":"/sota/natural-language-inference-on-rte","task":"Natural Language Inference","dataset":"RTE","model":"H3 125M (0-shot, rank classification)","rank_in_archive_order":88,"of":90,"metrics":{"Accuracy":"53.1%"},"uses_additional_data":false},{"leaderboard":"/sota/natural-language-inference-on-rte","task":"Natural Language Inference","dataset":"RTE","model":"H3 125M (3-shot, rank classification)","rank_in_archive_order":90,"of":90,"metrics":{"Accuracy":"52.3%"},"uses_additional_data":false},{"leaderboard":"/sota/question-answering-on-boolq","task":"Question Answering","dataset":"BoolQ","model":"Hybrid H3 1.3B (0-shot, logit scoring)","rank_in_archive_order":52,"of":65,"metrics":{"Accuracy":"61.7"},"uses_additional_data":false},{"leaderboard":"/sota/question-answering-on-boolq","task":"Question Answering","dataset":"BoolQ","model":"Hybrid H3 2.7B (3-shot, logit scoring)","rank_in_archive_order":55,"of":65,"metrics":{"Accuracy":"60.6"},"uses_additional_data":false},{"leaderboard":"/sota/question-answering-on-boolq","task":"Question Answering","dataset":"BoolQ","model":"Hybrid H3 125M (0-shot, logit scoring)","rank_in_archive_order":59,"of":65,"metrics":{"Accuracy":"59.6"},"uses_additional_data":false},{"leaderboard":"/sota/question-answering-on-boolq","task":"Question Answering","dataset":"BoolQ","model":"Hybrid H3 125M (3-shot, logit scoring)","rank_in_archive_order":61,"of":65,"metrics":{"Accuracy":"56.1"},"uses_additional_data":false},{"leaderboard":"/sota/question-answering-on-boolq","task":"Question Answering","dataset":"BoolQ","model":"Hybrid H3 125M (3-shot, rank classification)","rank_in_archive_order":62,"of":65,"metrics":{"Accuracy":"56.1"},"uses_additional_data":false},{"leaderboard":"/sota/question-answering-on-copa","task":"Question Answering","dataset":"COPA","model":"Hybrid H3 2.7B (0-shot, logit scoring)","rank_in_archive_order":35,"of":60,"metrics":{"Accuracy":"81"},"uses_additional_data":false},{"leaderboard":"/sota/question-answering-on-copa","task":"Question Answering","dataset":"COPA","model":"Hybrid H3 2.7B (3-shot, logit scoring)","rank_in_archive_order":42,"of":60,"metrics":{"Accuracy":"77"},"uses_additional_data":false},{"leaderboard":"/sota/question-answering-on-copa","task":"Question Answering","dataset":"COPA","model":"Hybrid H3 125M (0-shot, logit scoring)","rank_in_archive_order":51,"of":60,"metrics":{"Accuracy":"67"},"uses_additional_data":false},{"leaderboard":"/sota/question-answering-on-copa","task":"Question Answering","dataset":"COPA","model":"Hybrid H3 125M (0-shot, rank classification)","rank_in_archive_order":52,"of":60,"metrics":{"Accuracy":"67"},"uses_additional_data":false},{"leaderboard":"/sota/question-answering-on-copa","task":"Question Answering","dataset":"COPA","model":"H3 125M (0-shot, rank classification)","rank_in_archive_order":59,"of":60,"metrics":{"Accuracy":"51"},"uses_additional_data":false},{"leaderboard":"/sota/question-answering-on-multirc","task":"Question Answering","dataset":"MultiRC","model":"Hybrid H3 355M (3-shot, logit scoring)","rank_in_archive_order":27,"of":30,"metrics":{"EM":"59.7"},"uses_additional_data":false},{"leaderboard":"/sota/question-answering-on-multirc","task":"Question Answering","dataset":"MultiRC","model":"Hybrid H3 355M (0-shot, logit scoring)","rank_in_archive_order":28,"of":30,"metrics":{"EM":"59.5"},"uses_additional_data":false},{"leaderboard":"/sota/question-answering-on-multirc","task":"Question Answering","dataset":"MultiRC","model":"Hybrid H3 125M (0-shot, logit scoring)","rank_in_archive_order":29,"of":30,"metrics":{"EM":"51.4"},"uses_additional_data":false},{"leaderboard":"/sota/question-answering-on-multirc","task":"Question Answering","dataset":"MultiRC","model":"Hybrid H3 125M (3-shot, logit scoring)","rank_in_archive_order":30,"of":30,"metrics":{"EM":"48.9"},"uses_additional_data":false},{"leaderboard":"/sota/word-sense-disambiguation-on-words-in-context","task":"Word Sense Disambiguation","dataset":"Words in Context","model":"Hybrid H3 125M (0-shot, logit scoring)","rank_in_archive_order":29,"of":37,"metrics":{"Accuracy":"51.4"},"uses_additional_data":false},{"leaderboard":"/sota/word-sense-disambiguation-on-words-in-context","task":"Word Sense Disambiguation","dataset":"Words in Context","model":"Hybrid H3 125M (0-shot, rank classification)","rank_in_archive_order":30,"of":37,"metrics":{"Accuracy":"51.4"},"uses_additional_data":false},{"leaderboard":"/sota/word-sense-disambiguation-on-words-in-context","task":"Word Sense Disambiguation","dataset":"Words in Context","model":"Hybrid H3 125M (3-shot, logit scoring)","rank_in_archive_order":37,"of":37,"metrics":{"Accuracy":"49.1"},"uses_additional_data":false}],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2212.14052","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2212.14052"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/hazyresearch/safari","reach":{"status":"ok","spdx":"Apache-2.0"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/lindermanlab/S5","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/hazyresearch/h3","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran":3,"ran_fixture":1,"ran_draft_wrong":2,"ran_honours":1,"unverified":8},"by_repo_kind":{"official":{"samples":1,"ran":1,"repositories":1},"listed":{"samples":14,"ran":6,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"497dbc1b6483fce5","entry":"EMAKernel","repo":"hazyresearch/safari","repo_kind":"listed","path":"src/models/sequence/h3.py","file_url":"https://github.com/hazyresearch/safari/blob/HEAD/src/models/sequence/h3.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"497dbc1b6483fce5"}},{"code_sha256_prefix":"ca63d87a582adba1","entry":"SSKernelShift","repo":"hazyresearch/safari","repo_kind":"listed","path":"src/models/sequence/h3.py","file_url":"https://github.com/hazyresearch/safari/blob/HEAD/src/models/sequence/h3.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"ca63d87a582adba1"}},{"code_sha256_prefix":"5dd28d20c60f0c19","entry":"mul_sum","repo":"HazyResearch/H3","repo_kind":"official","path":"src/models/ssm/h3.py","file_url":"https://github.com/HazyResearch/H3/blob/HEAD/src/models/ssm/h3.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"5dd28d20c60f0c19"}},{"code_sha256_prefix":"cde6f4e2d1c79e3a","entry":"nplr","repo":"hazyresearch/safari","repo_kind":"listed","path":"src/models/sequence/h3.py","file_url":"https://github.com/hazyresearch/safari/blob/HEAD/src/models/sequence/h3.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"cde6f4e2d1c79e3a"}},{"code_sha256_prefix":"55a38587423268a5","entry":"power","repo":"hazyresearch/safari","repo_kind":"listed","path":"src/models/sequence/h3.py","file_url":"https://github.com/hazyresearch/safari/blob/HEAD/src/models/sequence/h3.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"55a38587423268a5"}},{"code_sha256_prefix":"7f226e8168260082","entry":"rank_correction","repo":"hazyresearch/safari","repo_kind":"listed","path":"src/models/sequence/h3.py","file_url":"https://github.com/hazyresearch/safari/blob/HEAD/src/models/sequence/h3.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"7f226e8168260082"}},{"code_sha256_prefix":"93675235174e1ee4","entry":"transition","repo":"hazyresearch/safari","repo_kind":"listed","path":"src/models/sequence/h3.py","file_url":"https://github.com/hazyresearch/safari/blob/HEAD/src/models/sequence/h3.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"93675235174e1ee4"}},{"code_sha256_prefix":"b100cfc9148d4a59","entry":"H3","repo":"hazyresearch/safari","repo_kind":"listed","path":"src/models/sequence/h3.py","file_url":"https://github.com/hazyresearch/safari/blob/HEAD/src/models/sequence/h3.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"b100cfc9148d4a59"}},{"code_sha256_prefix":"1d9eb34a580a6143","entry":"SSKernel","repo":"hazyresearch/safari","repo_kind":"listed","path":"src/models/sequence/h3.py","file_url":"https://github.com/hazyresearch/safari/blob/HEAD/src/models/sequence/h3.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"1d9eb34a580a6143"}},{"code_sha256_prefix":"60be78d8667c0da7","entry":"SSKernelDiag","repo":"hazyresearch/safari","repo_kind":"listed","path":"src/models/sequence/h3.py","file_url":"https://github.com/hazyresearch/safari/blob/HEAD/src/models/sequence/h3.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"60be78d8667c0da7"}},{"code_sha256_prefix":"c73230013cf47ccb","entry":"combination","repo":"hazyresearch/safari","repo_kind":"listed","path":"src/models/sequence/h3.py","file_url":"https://github.com/hazyresearch/safari/blob/HEAD/src/models/sequence/h3.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"c73230013cf47ccb"}},{"code_sha256_prefix":"871ae87aa2c2ffb6","entry":"dplr","repo":"hazyresearch/safari","repo_kind":"listed","path":"src/models/sequence/h3.py","file_url":"https://github.com/hazyresearch/safari/blob/HEAD/src/models/sequence/h3.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"871ae87aa2c2ffb6"}},{"code_sha256_prefix":"37bc70e26a2b9b9e","entry":"log_vandermonde","repo":"hazyresearch/safari","repo_kind":"listed","path":"src/models/sequence/h3.py","file_url":"https://github.com/hazyresearch/safari/blob/HEAD/src/models/sequence/h3.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"37bc70e26a2b9b9e"}},{"code_sha256_prefix":"2f58a2eaf4ce01c9","entry":"log_vandermonde_transpose","repo":"hazyresearch/safari","repo_kind":"listed","path":"src/models/sequence/h3.py","file_url":"https://github.com/hazyresearch/safari/blob/HEAD/src/models/sequence/h3.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"2f58a2eaf4ce01c9"}},{"code_sha256_prefix":"3d83038bb3d6c578","entry":"ssm","repo":"hazyresearch/safari","repo_kind":"listed","path":"src/models/sequence/h3.py","file_url":"https://github.com/hazyresearch/safari/blob/HEAD/src/models/sequence/h3.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"3d83038bb3d6c578"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}