{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/recurrentgemma-moving-past-transformers-for","title":"RecurrentGemma: Moving Past Transformers for Efficient Open Language Models","arxiv_id":"2404.07839","date":"2024-04-11","proceeding":null,"authors":["Aleksandar Botev","Soham De","Samuel L Smith","Anushan Fernando","George-Cristian Muraru","Ruba Haroun","Leonard Berrada","Razvan Pascanu","Pier Giuseppe Sessa","Robert Dadashi","Léonard Hussenot","Johan Ferret","Sertan Girgin","Olivier Bachem","Alek Andreev","Kathleen Kenealy","Thomas Mesnard","Cassidy Hardin","Surya Bhupatiraju","Shreya Pathak","Laurent SIfre","Morgane Rivière","Mihir Sanjay Kale","Juliette Love","Pouya Tafti","Armand Joulin","Noah Fiedel","Evan Senter","Yutian Chen","Srivatsan Srinivasan","Guillaume Desjardins","David Budden","Arnaud Doucet","Sharad Vikram","Adam Paszke","Trevor Gale","Sebastian Borgeaud","Charlie Chen","Andy Brock","Antonia Paterson","Jenny Brennan","Meg Risdal","Raj Gundluru","Nesh Devanathan","Paul Mooney","Nilay Chauhan","Phil Culliton","Luiz GUStavo Martins","Elisa Bandy","David Huntsperger","Glenn Cameron","Arthur Zucker","Tris Warkentin","Ludovic Peran","Minh Giang","Zoubin Ghahramani","Clément Farabet","Koray Kavukcuoglu","Demis Hassabis","Raia Hadsell","Yee Whye Teh","Nando de Frietas"],"abstract":"We introduce RecurrentGemma, a family of open language models which uses Google's novel Griffin architecture. Griffin combines linear recurrences with local attention to achieve excellent performance on language. It has a fixed-sized state, which reduces memory use and enables efficient inference on long sequences. We provide two sizes of models, containing 2B and 9B parameters, and provide pre-trained and instruction tuned variants for both. Our models achieve comparable performance to similarly-sized Gemma baselines despite being trained on fewer tokens.","url_abs":"https://arxiv.org/abs/2404.07839v2","url_pdf":"https://arxiv.org/pdf/2404.07839v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"recurrentgemma-moving-past-transformers-for","repo_url":"https://github.com/google-deepmind/recurrentgemma","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"jax","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"language-modelling","task_name":"Language Modelling"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2404.07839","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2404.07839"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/google-deepmind/recurrentgemma","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran":8,"unverified":3},"by_repo_kind":{"official":{"samples":11,"ran":8,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"4871f51f7f0bd2b1","entry":"apply_it_formatter","repo":"google-deepmind/recurrentgemma","repo_kind":"official","path":"recurrentgemma/common.py","file_url":"https://github.com/google-deepmind/recurrentgemma/blob/HEAD/recurrentgemma/common.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"4871f51f7f0bd2b1"}},{"code_sha256_prefix":"e3ba522a925bb045","entry":"flatten_nested_dict","repo":"google-deepmind/recurrentgemma","repo_kind":"official","path":"recurrentgemma/conversion.py","file_url":"https://github.com/google-deepmind/recurrentgemma/blob/HEAD/recurrentgemma/conversion.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"e3ba522a925bb045"}},{"code_sha256_prefix":"dd656979e56353bf","entry":"gelu","repo":"google-deepmind/recurrentgemma","repo_kind":"official","path":"recurrentgemma/torch/modules.py","file_url":"https://github.com/google-deepmind/recurrentgemma/blob/HEAD/recurrentgemma/torch/modules.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"dd656979e56353bf"}},{"code_sha256_prefix":"f37defe489946350","entry":"get_num_seq_shards","repo":"google-deepmind/recurrentgemma","repo_kind":"official","path":"recurrentgemma/jax/pallas.py","file_url":"https://github.com/google-deepmind/recurrentgemma/blob/HEAD/recurrentgemma/jax/pallas.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"f37defe489946350"}},{"code_sha256_prefix":"e54de79948085b3e","entry":"jax_array_to_torch_tensor","repo":"google-deepmind/recurrentgemma","repo_kind":"official","path":"recurrentgemma/conversion.py","file_url":"https://github.com/google-deepmind/recurrentgemma/blob/HEAD/recurrentgemma/conversion.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"e54de79948085b3e"}},{"code_sha256_prefix":"bbdb17f3635afb99","entry":"rnn_param_init","repo":"google-deepmind/recurrentgemma","repo_kind":"official","path":"recurrentgemma/torch/layers.py","file_url":"https://github.com/google-deepmind/recurrentgemma/blob/HEAD/recurrentgemma/torch/layers.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"bbdb17f3635afb99"}},{"code_sha256_prefix":"54240e1af7110f38","entry":"sequence_shard_index","repo":"google-deepmind/recurrentgemma","repo_kind":"official","path":"recurrentgemma/jax/pallas.py","file_url":"https://github.com/google-deepmind/recurrentgemma/blob/HEAD/recurrentgemma/jax/pallas.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"54240e1af7110f38"}},{"code_sha256_prefix":"e0bcd16aebc4a14d","entry":"torch_tensor_to_jax_array","repo":"google-deepmind/recurrentgemma","repo_kind":"official","path":"recurrentgemma/conversion.py","file_url":"https://github.com/google-deepmind/recurrentgemma/blob/HEAD/recurrentgemma/conversion.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"e0bcd16aebc4a14d"}},{"code_sha256_prefix":"976bdf5aa58257cb","entry":"sigmoid","repo":"google-deepmind/recurrentgemma","repo_kind":"official","path":"recurrentgemma/jax/complex_lib.py","file_url":"https://github.com/google-deepmind/recurrentgemma/blob/HEAD/recurrentgemma/jax/complex_lib.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"976bdf5aa58257cb"}},{"code_sha256_prefix":"18974767a2721a73","entry":"softplus","repo":"google-deepmind/recurrentgemma","repo_kind":"official","path":"recurrentgemma/jax/complex_lib.py","file_url":"https://github.com/google-deepmind/recurrentgemma/blob/HEAD/recurrentgemma/jax/complex_lib.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"18974767a2721a73"}},{"code_sha256_prefix":"72da31023860da4c","entry":"to_custom_complex","repo":"google-deepmind/recurrentgemma","repo_kind":"official","path":"recurrentgemma/jax/complex_lib.py","file_url":"https://github.com/google-deepmind/recurrentgemma/blob/HEAD/recurrentgemma/jax/complex_lib.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"72da31023860da4c"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}