{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/goldfinch-high-performance-rwkv-transformer","title":"GoldFinch: High Performance RWKV/Transformer Hybrid with Linear Pre-Fill and Extreme KV-Cache Compression","arxiv_id":"2407.12077","date":"2024-07-16","proceeding":null,"authors":["Daniel Goldstein","Fares Obeid","Eric Alcaide","Guangyu Song","Eugene Cheah"],"abstract":"We introduce GoldFinch, a hybrid Linear Attention/Transformer sequence model that uses a new technique to efficiently generate a highly compressed and reusable KV-Cache in linear time and space with respect to sequence length. GoldFinch stacks our new GOLD transformer on top of an enhanced version of the Finch (RWKV-6) architecture. We train up to 1.5B parameter class models of the Finch, Llama, and GoldFinch architectures, and find dramatically improved modeling performance relative to both Finch and Llama. Our cache size savings increase linearly with model layer count, ranging from 756-2550 times smaller than the traditional transformer cache for common sizes, enabling inference of extremely large context lengths even on limited hardware. Although autoregressive generation has O(n) time complexity per token because of attention, pre-fill computation of the entire initial cache state for a submitted context costs only O(1) time per token due to the use of a recurrent neural network (RNN) to generate this cache. We release our trained weights and training code under the Apache 2.0 license for community use.","url_abs":"https://arxiv.org/abs/2407.12077v1","url_pdf":"https://arxiv.org/pdf/2407.12077v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"goldfinch-high-performance-rwkv-transformer","repo_url":"https://github.com/SmerkyG/GoldFinch-paper","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"goldfinch-high-performance-rwkv-transformer","repo_url":"https://github.com/recursal/GoldFinch-paper","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[],"methods":[{"method_slug":"finch","method_name":"FINCH"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2407.12077","atlas_url":"https://app.syntology.ai/?focus=2407.12077","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2407.12077"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/recursal/GoldFinch-paper","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/SmerkyG/GoldFinch-paper","reach":null}],"summary":{"ran_honours":2,"ran_violates":1,"ran_draft_wrong":2},"by_repo_kind":{"official":{"samples":5,"ran":5,"repositories":2}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"748c47730233b5bd","entry":"alibi_mask","repo":"recursal/GoldFinch-paper","repo_kind":"official","path":"src/model.py","file_url":"https://github.com/recursal/GoldFinch-paper/blob/HEAD/src/model.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"748c47730233b5bd"}},{"code_sha256_prefix":"9a702f2c1e793486","entry":"causal_bias_mask","repo":"recursal/GoldFinch-paper","repo_kind":"official","path":"src/model.py","file_url":"https://github.com/recursal/GoldFinch-paper/blob/HEAD/src/model.py","link_basis":"first_harvest_node","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"9a702f2c1e793486"}},{"code_sha256_prefix":"e8df29c14266e90d","entry":"convert_dict_to_config","repo":"SmerkyG/GoldFinch-paper","repo_kind":"official","path":"configs.py","file_url":"https://github.com/SmerkyG/GoldFinch-paper/blob/HEAD/configs.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"e8df29c14266e90d"}},{"code_sha256_prefix":"332d7941bf886d82","entry":"literal_eval","repo":"SmerkyG/GoldFinch-paper","repo_kind":"official","path":"configs.py","file_url":"https://github.com/SmerkyG/GoldFinch-paper/blob/HEAD/configs.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"332d7941bf886d82"}},{"code_sha256_prefix":"9c3d0d0b3af30bb8","entry":"merge_config","repo":"SmerkyG/GoldFinch-paper","repo_kind":"official","path":"configs.py","file_url":"https://github.com/SmerkyG/GoldFinch-paper/blob/HEAD/configs.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"9c3d0d0b3af30bb8"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}