{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/dnabert-2-efficient-foundation-model-and","title":"DNABERT-2: Efficient Foundation Model and Benchmark For Multi-Species Genome","arxiv_id":"2306.15006","date":"2023-06-26","proceeding":null,"authors":["Zhihan Zhou","Yanrong Ji","Weijian Li","Pratik Dutta","Ramana Davuluri","Han Liu"],"abstract":"Decoding the linguistic intricacies of the genome is a crucial problem in biology, and pre-trained foundational models such as DNABERT and Nucleotide Transformer have made significant strides in this area. Existing works have largely hinged on k-mer, fixed-length permutations of A, T, C, and G, as the token of the genome language due to its simplicity. However, we argue that the computation and sample inefficiencies introduced by k-mer tokenization are primary obstacles in developing large genome foundational models. We provide conceptual and empirical insights into genome tokenization, building on which we propose to replace k-mer tokenization with Byte Pair Encoding (BPE), a statistics-based data compression algorithm that constructs tokens by iteratively merging the most frequent co-occurring genome segment in the corpus. We demonstrate that BPE not only overcomes the limitations of k-mer tokenization but also benefits from the computational efficiency of non-overlapping tokenization. Based on these insights, we introduce DNABERT-2, a refined genome foundation model that adapts an efficient tokenizer and employs multiple strategies to overcome input length constraints, reduce time and memory expenditure, and enhance model capability. Furthermore, we identify the absence of a comprehensive and standardized benchmark for genome understanding as another significant impediment to fair comparative analysis. In response, we propose the Genome Understanding Evaluation (GUE), a comprehensive multi-species genome classification dataset that amalgamates $36$ distinct datasets across $9$ tasks, with input lengths ranging from $70$ to $10000$. Through comprehensive experiments on the GUE benchmark, we demonstrate that DNABERT-2 achieves comparable performance to the state-of-the-art model with $21 \\times$ fewer parameters and approximately $92 \\times$ less GPU time in pre-training.","url_abs":"https://arxiv.org/abs/2306.15006v2","url_pdf":"https://arxiv.org/pdf/2306.15006v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"dnabert-2-efficient-foundation-model-and","repo_url":"https://github.com/magics-lab/dnabert_2","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}},{"paper_slug":"dnabert-2-efficient-foundation-model-and","repo_url":"https://github.com/zhihan1996/dnabert_2","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}},{"paper_slug":"dnabert-2-efficient-foundation-model-and","repo_url":"https://github.com/frederikkemarin/bend","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"BSD-3-Clause"}},{"paper_slug":"dnabert-2-efficient-foundation-model-and","repo_url":"https://github.com/jerryji1993/dnabert","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}},{"paper_slug":"dnabert-2-efficient-foundation-model-and","repo_url":"https://github.com/jimmylihui/OpenGenome","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"dnabert-2-efficient-foundation-model-and","repo_url":"https://github.com/jimmylihui/genbench","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"computational-efficiency","task_name":"Computational Efficiency"},{"task_slug":"core-promoter-detection","task_name":"Core Promoter Detection"},{"task_slug":"covid-variant-prediction","task_name":"Covid Variant Prediction"},{"task_slug":"dna-analysis","task_name":"DNA analysis"},{"task_slug":"data-compression","task_name":"Data Compression"},{"task_slug":"epigenetic-marks-prediction","task_name":"Epigenetic Marks Prediction"},{"task_slug":null,"task_name":"GPU"},{"task_slug":"genome-understanding","task_name":"Genome Understanding"},{"task_slug":"promoter-detection","task_name":"Promoter Detection"},{"task_slug":"splice-site-prediction","task_name":"Splice Site Prediction"},{"task_slug":"transcription-factor-binding-site-prediction","task_name":"Transcription Factor Binding Site Prediction"},{"task_slug":"transcription-factor-binding-site-prediction-1","task_name":"Transcription Factor Binding Site Prediction (Human)"},{"task_slug":"transcription-factor-binding-site-prediction-2","task_name":"Transcription Factor Binding Site Prediction (Mouse)"}],"methods":[{"method_slug":"absolute-position-encodings","method_name":"Absolute Position Encodings"},{"method_slug":"adam","method_name":"Adam"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"bpe","method_name":"BPE"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"label-smoothing","method_name":"Label Smoothing"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"position-wise-feed-forward-layer","method_name":"Position-Wise Feed-Forward Layer"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"transformer","method_name":"Transformer"}],"datasets_introduced":[{"slug":"gue","name":"GUE","full_name":"Genome Understanding Evaluation"}],"methods_introduced":[],"results":[{"leaderboard":"/sota/core-promoter-detection-on-gue","task":"Core Promoter Detection","dataset":"GUE","model":"DNABERT-2-117M","rank_in_archive_order":1,"of":1,"metrics":{"MCC":"70.52"},"uses_additional_data":false},{"leaderboard":"/sota/covid-variant-prediction-on-gue","task":"Covid Variant Prediction","dataset":"GUE","model":"DNABERT-2-117M","rank_in_archive_order":1,"of":1,"metrics":{"Avg F1":"71.02"},"uses_additional_data":false},{"leaderboard":"/sota/epigenetic-marks-prediction-on-gue","task":"Epigenetic Marks Prediction","dataset":"GUE","model":"DNABERT-2-117M","rank_in_archive_order":1,"of":1,"metrics":{"MCC":"55.98"},"uses_additional_data":false},{"leaderboard":"/sota/promoter-detection-on-gue","task":"Promoter Detection","dataset":"GUE","model":"DNABERT-2-117M","rank_in_archive_order":1,"of":1,"metrics":{"MCC":"84.21"},"uses_additional_data":false},{"leaderboard":"/sota/splice-site-prediction-on-gue","task":"Splice Site Prediction","dataset":"GUE","model":"DNABERT-2-117M","rank_in_archive_order":1,"of":1,"metrics":{"MCC":"84.99"},"uses_additional_data":false},{"leaderboard":"/sota/transcription-factor-binding-site-prediction","task":"Transcription Factor Binding Site Prediction (Human)","dataset":"GUE","model":"DNABERT-2-117M","rank_in_archive_order":1,"of":1,"metrics":{"MCC":"70.10"},"uses_additional_data":false},{"leaderboard":"/sota/transcription-factor-binding-site-prediction-1","task":"Transcription Factor Binding Site Prediction (Mouse)","dataset":"GUE","model":"DNABERT-2-117M","rank_in_archive_order":1,"of":1,"metrics":{"MCC":"67.99"},"uses_additional_data":false}],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2306.15006","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2306.15006"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/MAGICS-LAB/DNABERT_2","reach":{"status":"ok","spdx":"Apache-2.0"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/magics-lab/dnabert_2","reach":{"status":"ok","spdx":"Apache-2.0"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/jimmylihui/genbench","reach":{"status":"ok","spdx":"Apache-2.0"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/zhihan1996/dnabert_2","reach":{"status":"ok","spdx":"Apache-2.0"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/frederikkemarin/bend","reach":{"status":"ok","spdx":"BSD-3-Clause"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/jimmylihui/OpenGenome","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/jerryji1993/dnabert","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran_draft_wrong":3,"ran":3,"ran_honours":1,"ran_fixture":1,"unverified":15},"by_repo_kind":{"listed":{"samples":23,"ran":8,"repositories":2}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"0f786c407fb1ee4c","entry":"swish","repo":"jerryji1993/dnabert","repo_kind":"listed","path":"src/transformers/activations.py","file_url":"https://github.com/jerryji1993/dnabert/blob/HEAD/src/transformers/activations.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":2,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"0f786c407fb1ee4c"}},{"code_sha256_prefix":"f1c7f50e43f8c65f","entry":"BertGatedLinearUnitMLP","repo":"frederikkemarin/bend","repo_kind":"listed","path":"bend/utils/embedders.py","file_url":"https://github.com/frederikkemarin/bend/blob/HEAD/bend/utils/embedders.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"f1c7f50e43f8c65f"}},{"code_sha256_prefix":"82086191fd1b0dae","entry":"BertPooler","repo":"frederikkemarin/bend","repo_kind":"listed","path":"bend/utils/embedders.py","file_url":"https://github.com/frederikkemarin/bend/blob/HEAD/bend/utils/embedders.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"82086191fd1b0dae"}},{"code_sha256_prefix":"4c10b29b9932aaac","entry":"BertPredictionHeadTransform","repo":"frederikkemarin/bend","repo_kind":"listed","path":"bend/utils/embedders.py","file_url":"https://github.com/frederikkemarin/bend/blob/HEAD/bend/utils/embedders.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"4c10b29b9932aaac"}},{"code_sha256_prefix":"77601724cad03f95","entry":"gelu_new","repo":"jerryji1993/dnabert","repo_kind":"listed","path":"src/transformers/activations.py","file_url":"https://github.com/jerryji1993/dnabert/blob/HEAD/src/transformers/activations.py","link_basis":"harvester_set","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"77601724cad03f95"}},{"code_sha256_prefix":"dbf4ecb6112f8eef","entry":"pad_input","repo":"frederikkemarin/bend","repo_kind":"listed","path":"bend/utils/embedders.py","file_url":"https://github.com/frederikkemarin/bend/blob/HEAD/bend/utils/embedders.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"dbf4ecb6112f8eef"}},{"code_sha256_prefix":"e819e759304fc70a","entry":"unpad_input","repo":"frederikkemarin/bend","repo_kind":"listed","path":"bend/utils/embedders.py","file_url":"https://github.com/frederikkemarin/bend/blob/HEAD/bend/utils/embedders.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"e819e759304fc70a"}},{"code_sha256_prefix":"9c99ec96528c0f00","entry":"unpad_input_only","repo":"frederikkemarin/bend","repo_kind":"listed","path":"bend/utils/embedders.py","file_url":"https://github.com/frederikkemarin/bend/blob/HEAD/bend/utils/embedders.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"9c99ec96528c0f00"}},{"code_sha256_prefix":"48ac189cd4db0ad9","entry":"BertEmbeddings","repo":"frederikkemarin/bend","repo_kind":"listed","path":"bend/utils/embedders.py","file_url":"https://github.com/frederikkemarin/bend/blob/HEAD/bend/utils/embedders.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"48ac189cd4db0ad9"}},{"code_sha256_prefix":"3386604c83d5235b","entry":"BertEncoder","repo":"frederikkemarin/bend","repo_kind":"listed","path":"bend/utils/embedders.py","file_url":"https://github.com/frederikkemarin/bend/blob/HEAD/bend/utils/embedders.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"3386604c83d5235b"}},{"code_sha256_prefix":"810c58679e40ef3e","entry":"BertLMPredictionHead","repo":"frederikkemarin/bend","repo_kind":"listed","path":"bend/utils/embedders.py","file_url":"https://github.com/frederikkemarin/bend/blob/HEAD/bend/utils/embedders.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"810c58679e40ef3e"}},{"code_sha256_prefix":"f82f6d515d78b9fa","entry":"BertLayer","repo":"frederikkemarin/bend","repo_kind":"listed","path":"bend/utils/embedders.py","file_url":"https://github.com/frederikkemarin/bend/blob/HEAD/bend/utils/embedders.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"f82f6d515d78b9fa"}},{"code_sha256_prefix":"f90d502733dc3fc8","entry":"BertModel","repo":"frederikkemarin/bend","repo_kind":"listed","path":"bend/utils/embedders.py","file_url":"https://github.com/frederikkemarin/bend/blob/HEAD/bend/utils/embedders.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"f90d502733dc3fc8"}},{"code_sha256_prefix":"d72aad474f62da31","entry":"BertOnlyMLMHead","repo":"frederikkemarin/bend","repo_kind":"listed","path":"bend/utils/embedders.py","file_url":"https://github.com/frederikkemarin/bend/blob/HEAD/bend/utils/embedders.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"d72aad474f62da31"}},{"code_sha256_prefix":"ce7a1b8194d588f0","entry":"BertUnpadAttention","repo":"frederikkemarin/bend","repo_kind":"listed","path":"bend/utils/embedders.py","file_url":"https://github.com/frederikkemarin/bend/blob/HEAD/bend/utils/embedders.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"ce7a1b8194d588f0"}},{"code_sha256_prefix":"6957915768d8a616","entry":"BertUnpadSelfAttention","repo":"frederikkemarin/bend","repo_kind":"listed","path":"bend/utils/embedders.py","file_url":"https://github.com/frederikkemarin/bend/blob/HEAD/bend/utils/embedders.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"6957915768d8a616"}},{"code_sha256_prefix":"c4e9c025a630c701","entry":"DNABert2Embedder","repo":"frederikkemarin/bend","repo_kind":"listed","path":"bend/utils/embedders.py","file_url":"https://github.com/frederikkemarin/bend/blob/HEAD/bend/utils/embedders.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"c4e9c025a630c701"}},{"code_sha256_prefix":"2f4b4de6eea982fe","entry":"IndexFirstAxis","repo":"frederikkemarin/bend","repo_kind":"listed","path":"bend/utils/embedders.py","file_url":"https://github.com/frederikkemarin/bend/blob/HEAD/bend/utils/embedders.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"2f4b4de6eea982fe"}},{"code_sha256_prefix":"0e02bf4a27dbaf92","entry":"contiguous_regions","repo":"jerryji1993/dnabert","repo_kind":"listed","path":"motif/motif_utils.py","file_url":"https://github.com/jerryji1993/dnabert/blob/HEAD/motif/motif_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"0e02bf4a27dbaf92"}},{"code_sha256_prefix":"c91ea05b2f9c818a","entry":"get_activation","repo":"jerryji1993/dnabert","repo_kind":"listed","path":"src/transformers/activations.py","file_url":"https://github.com/jerryji1993/dnabert/blob/HEAD/src/transformers/activations.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"c91ea05b2f9c818a"}},{"code_sha256_prefix":"2f94f1febf69f788","entry":"kmer2seq","repo":"jerryji1993/dnabert","repo_kind":"listed","path":"motif/motif_utils.py","file_url":"https://github.com/jerryji1993/dnabert/blob/HEAD/motif/motif_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"2f94f1febf69f788"}},{"code_sha256_prefix":"97526e4fc5605ed7","entry":"mutate","repo":"jerryji1993/dnabert","repo_kind":"listed","path":"SNP/mutate_seqs.py","file_url":"https://github.com/jerryji1993/dnabert/blob/HEAD/SNP/mutate_seqs.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"97526e4fc5605ed7"}},{"code_sha256_prefix":"86ed11d1b2a6f597","entry":"seq2kmer","repo":"jerryji1993/dnabert","repo_kind":"listed","path":"motif/motif_utils.py","file_url":"https://github.com/jerryji1993/dnabert/blob/HEAD/motif/motif_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"86ed11d1b2a6f597"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}