{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/janusdna-a-powerful-bi-directional-hybrid-dna","title":"JanusDNA: A Powerful Bi-directional Hybrid DNA Foundation Model","arxiv_id":"2505.17257","date":"2025-05-22","proceeding":null,"authors":["Qihao Duan","Bingding Huang","Zhenqiao Song","Irina Lehmann","Lei Gu","Roland Eils","Benjamin Wild"],"abstract":"Large language models (LLMs) have revolutionized natural language processing and are increasingly applied to other sequential data types, including genetic sequences. However, adapting LLMs to genomics presents significant challenges. Capturing complex genomic interactions requires modeling long-range dependencies within DNA sequences, where interactions often span over 10,000 base pairs, even within a single gene, posing substantial computational burdens under conventional model architectures and training paradigms. Moreover, standard LLM training approaches are suboptimal for DNA: autoregressive training, while efficient, supports only unidirectional understanding. However, DNA is inherently bidirectional, e.g., bidirectional promoters regulate transcription in both directions and account for nearly 11% of human gene expression. Masked language models (MLMs) allow bidirectional understanding but are inefficient, as only masked tokens contribute to the loss per step. To address these limitations, we introduce JanusDNA, the first bidirectional DNA foundation model built upon a novel pretraining paradigm that combines the optimization efficiency of autoregressive modeling with the bidirectional comprehension of masked modeling. JanusDNA adopts a hybrid Mamba, Attention and Mixture of Experts (MoE) architecture, combining long-range modeling of Attention with efficient sequential learning of Mamba. MoE layers further scale model capacity via sparse activation while keeping computational cost low. Notably, JanusDNA processes up to 1 million base pairs at single nucleotide resolution on a single 80GB GPU. Extensive experiments and ablations show JanusDNA achieves new SOTA results on three genomic representation benchmarks, outperforming models with 250x more activated parameters. Code: https://github.com/Qihao-Duan/JanusDNA","url_abs":"https://arxiv.org/abs/2505.17257v2","url_pdf":"https://arxiv.org/pdf/2505.17257v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"janusdna-a-powerful-bi-directional-hybrid-dna","repo_url":"https://github.com/qihao-duan/janusdna","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":null,"task_name":"GPU"},{"task_slug":"long-range-modeling","task_name":"Long-range modeling"},{"task_slug":"mamba","task_name":"Mamba"},{"task_slug":"mixture-of-experts","task_name":"Mixture-of-Experts"}],"methods":[{"method_slug":"attention","method_name":"Attention"},{"method_slug":"base","method_name":"BASE"},{"method_slug":"mamba","method_name":"Mamba"},{"method_slug":"moe","method_name":"MoE"},{"method_slug":"softmax","method_name":"Softmax"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2505.17257","atlas_url":"https://app.syntology.ai/?focus=2505.17257","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2505.17257"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/Qihao-Duan/JanusDNA","reach":{"status":"ok","spdx":"Apache-2.0"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/qihao-duan/janusdna","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran":3,"ran_fixture":2},"by_repo_kind":{"official":{"samples":5,"ran":5,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"fdcd79ecc366f956","entry":"compute_expert_counts","repo":"Qihao-Duan/JanusDNA","repo_kind":"official","path":"janusdna/modeling_janusdna.py","file_url":"https://github.com/Qihao-Duan/JanusDNA/blob/HEAD/janusdna/modeling_janusdna.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"fdcd79ecc366f956"}},{"code_sha256_prefix":"83de732cfd3e479f","entry":"cross_entropy","repo":"Qihao-Duan/JanusDNA","repo_kind":"official","path":"caduceus/modeling_caduceus.py","file_url":"https://github.com/Qihao-Duan/JanusDNA/blob/HEAD/caduceus/modeling_caduceus.py","link_basis":"harvester_set","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"83de732cfd3e479f"}},{"code_sha256_prefix":"afa4dd8b92112788","entry":"load_balancing_loss_func","repo":"Qihao-Duan/JanusDNA","repo_kind":"official","path":"janusdna/modeling_janusdna.py","file_url":"https://github.com/Qihao-Duan/JanusDNA/blob/HEAD/janusdna/modeling_janusdna.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"afa4dd8b92112788"}},{"code_sha256_prefix":"74a4a9e0f04efa01","entry":"rank_zero_experiment","repo":"Qihao-Duan/JanusDNA","repo_kind":"official","path":"evaluation.py","file_url":"https://github.com/Qihao-Duan/JanusDNA/blob/HEAD/evaluation.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"74a4a9e0f04efa01"}},{"code_sha256_prefix":"2ef47d00e83953ae","entry":"weighted_cross_entropy","repo":"Qihao-Duan/JanusDNA","repo_kind":"official","path":"caduceus/modeling_caduceus.py","file_url":"https://github.com/Qihao-Duan/JanusDNA/blob/HEAD/caduceus/modeling_caduceus.py","link_basis":"harvester_set","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"2ef47d00e83953ae"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}