{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/rinalmo-general-purpose-rna-language-models","title":"RiNALMo: General-Purpose RNA Language Models Can Generalize Well on Structure Prediction Tasks","arxiv_id":"2403.00043","date":"2024-02-29","proceeding":null,"authors":["Rafael Josip Penić","Tin Vlašić","Roland G. Huber","Yue Wan","Mile Šikić"],"abstract":"While RNA has recently been recognized as an interesting small-molecule drug target, many challenges remain to be addressed before we take full advantage of it. This emphasizes the necessity to improve our understanding of its structures and functions. Over the years, sequencing technologies have produced an enormous amount of unlabeled RNA data, which hides a huge potential. Motivated by the successes of protein language models, we introduce RiboNucleic Acid Language Model (RiNALMo) to unveil the hidden code of RNA. RiNALMo is the largest RNA language model to date, with 650M parameters pre-trained on 36M non-coding RNA sequences from several databases. It can extract hidden knowledge and capture the underlying structure information implicitly embedded within the RNA sequences. RiNALMo achieves state-of-the-art results on several downstream tasks. Notably, we show that its generalization capabilities overcome the inability of other deep learning methods for secondary structure prediction to generalize on unseen RNA families.","url_abs":"https://arxiv.org/abs/2403.00043v2","url_pdf":"https://arxiv.org/pdf/2403.00043v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"rinalmo-general-purpose-rna-language-models","repo_url":"https://github.com/lbcb-sci/rinalmo","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}},{"paper_slug":"rinalmo-general-purpose-rna-language-models","repo_url":"https://github.com/ml4bio/rna-fm","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"language-modeling","task_name":"Language Modeling"},{"task_slug":"language-modelling","task_name":"Language Modelling"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2403.00043","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2403.00043"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/lbcb-sci/rinalmo","reach":{"status":"ok","spdx":"Apache-2.0"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/ml4bio/rna-fm","reach":null}],"summary":{"ran_violates":1,"ran":2,"ran_honours":1,"unverified":1},"by_repo_kind":{"official":{"samples":5,"ran":4,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"986a45db92ff30fe","entry":"apply_rotary_pos_emb","repo":"lbcb-sci/rinalmo","repo_kind":"official","path":"rinalmo/model/rope.py","file_url":"https://github.com/lbcb-sci/rinalmo/blob/HEAD/rinalmo/model/rope.py","link_basis":"plan_row","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"986a45db92ff30fe"}},{"code_sha256_prefix":"f6afd2235cdb7855","entry":"dot_product_attention","repo":"lbcb-sci/rinalmo","repo_kind":"official","path":"rinalmo/model/attention.py","file_url":"https://github.com/lbcb-sci/rinalmo/blob/HEAD/rinalmo/model/attention.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"f6afd2235cdb7855"}},{"code_sha256_prefix":"68ea47ffa17f6d99","entry":"is_canonical_sequence","repo":"lbcb-sci/rinalmo","repo_kind":"official","path":"rinalmo/utils/prepare_ncrna_classification_data.py","file_url":"https://github.com/lbcb-sci/rinalmo/blob/HEAD/rinalmo/utils/prepare_ncrna_classification_data.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"68ea47ffa17f6d99"}},{"code_sha256_prefix":"437c7011506995a3","entry":"rotate_half","repo":"lbcb-sci/rinalmo","repo_kind":"official","path":"rinalmo/model/rope.py","file_url":"https://github.com/lbcb-sci/rinalmo/blob/HEAD/rinalmo/model/rope.py","link_basis":"harvester_set","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"437c7011506995a3"}},{"code_sha256_prefix":"b54f0f8fcde6f675","entry":"download_splice_site_data","repo":"lbcb-sci/rinalmo","repo_kind":"official","path":"rinalmo/utils/download.py","file_url":"https://github.com/lbcb-sci/rinalmo/blob/HEAD/rinalmo/utils/download.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"b54f0f8fcde6f675"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}