{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/barcodebert-transformers-for-biodiversity","title":"BarcodeBERT: Transformers for Biodiversity Analysis","arxiv_id":"2311.02401","date":"2023-11-04","proceeding":null,"authors":["Pablo Millan Arias","Niousha Sadjadi","Monireh Safari","ZeMing Gong","Austin T. Wang","Joakim Bruslund Haurum","Iuliia Zarubiieva","Dirk Steinke","Lila Kari","Angel X. Chang","Scott C. Lowe","Graham W. Taylor"],"abstract":"In the global challenge of understanding and characterizing biodiversity, short species-specific genomic sequences known as DNA barcodes play a critical role, enabling fine-grained comparisons among organisms within the same kingdom of life. Although machine learning algorithms specifically designed for the analysis of DNA barcodes are becoming more popular, most existing methodologies rely on generic supervised training algorithms. We introduce BarcodeBERT, a family of models tailored to biodiversity analysis and trained exclusively on data from a reference library of 1.5M invertebrate DNA barcodes. We compared the performance of BarcodeBERT on taxonomic identification tasks against a spectrum of machine learning approaches including supervised training of classical neural architectures and fine-tuning of general DNA foundation models. Our self-supervised pretraining strategies on domain-specific data outperform fine-tuned foundation models, especially in identification tasks involving lower taxa such as genera and species. We also compared BarcodeBERT with BLAST, one of the most widely used bioinformatics tools for sequence searching, and found that our method matched BLAST's performance in species-level classification while being 55 times faster. Our analysis of masking and tokenization strategies also provides practical guidance for building customized DNA language models, emphasizing the importance of aligning model training strategies with dataset characteristics and domain knowledge. The code repository is available at https://github.com/bioscan-ml/BarcodeBERT.","url_abs":"https://arxiv.org/abs/2311.02401v2","url_pdf":"https://arxiv.org/pdf/2311.02401v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"barcodebert-transformers-for-biodiversity","repo_url":"https://github.com/bioscan-ml/barcodebert","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"barcodebert-transformers-for-biodiversity","repo_url":"https://github.com/kari-genomics-lab/barcodebert","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"barcodebert-transformers-for-biodiversity","repo_url":"https://github.com/bioscan-ml/bioscan-5m","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":null},{"paper_slug":"barcodebert-transformers-for-biodiversity","repo_url":"https://github.com/zahrag/BIOSCAN-5M","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok","spdx":"NOASSERTION"}}],"tasks":[{"task_slug":"model-selection","task_name":"Model Selection"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2311.02401","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2311.02401"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/bioscan-ml/BarcodeBERT","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/zahrag/BIOSCAN-5M","reach":{"status":"ok","spdx":"NOASSERTION"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/bioscan-ml/barcodebert","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/bioscan-ml/bioscan-5m","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/kari-genomics-lab/barcodebert","reach":null}],"summary":{"ran":11,"ran_draft_wrong":1,"unverified":4},"by_repo_kind":{"official":{"samples":16,"ran":12,"repositories":2}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"5131da2c2a95f44d","entry":"KmerTokenizer","repo":"kari-genomics-lab/barcodebert","repo_kind":"official","path":"barcodebert/bzsl/models/model.py","file_url":"https://github.com/kari-genomics-lab/barcodebert/blob/HEAD/barcodebert/bzsl/models/model.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"5131da2c2a95f44d"}},{"code_sha256_prefix":"095f7547ac0cb25e","entry":"categorical_cross_entropy","repo":"bioscan-ml/BarcodeBERT","repo_kind":"official","path":"barcodebert/bzsl/finetuning/supervised_learning.py","file_url":"https://github.com/bioscan-ml/BarcodeBERT/blob/HEAD/barcodebert/bzsl/finetuning/supervised_learning.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"095f7547ac0cb25e"}},{"code_sha256_prefix":"db3d69966ddebb8f","entry":"construct_dataloader","repo":"bioscan-ml/BarcodeBERT","repo_kind":"official","path":"barcodebert/bzsl/finetuning/supervised_learning.py","file_url":"https://github.com/bioscan-ml/BarcodeBERT/blob/HEAD/barcodebert/bzsl/finetuning/supervised_learning.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"db3d69966ddebb8f"}},{"code_sha256_prefix":"da25bbcd7de8221b","entry":"determine_epoch_seed","repo":"bioscan-ml/BarcodeBERT","repo_kind":"official","path":"barcodebert/utils.py","file_url":"https://github.com/bioscan-ml/BarcodeBERT/blob/HEAD/barcodebert/utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"da25bbcd7de8221b"}},{"code_sha256_prefix":"38d6d30dfe22d0ed","entry":"evaluate","repo":"bioscan-ml/barcodebert","repo_kind":"official","path":"barcodebert/pretraining.py","file_url":"https://github.com/bioscan-ml/barcodebert/blob/HEAD/barcodebert/pretraining.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"38d6d30dfe22d0ed"}},{"code_sha256_prefix":"eba6034b65efb5e6","entry":"evaluate","repo":"bioscan-ml/BarcodeBERT","repo_kind":"official","path":"barcodebert/evaluation.py","file_url":"https://github.com/bioscan-ml/BarcodeBERT/blob/HEAD/barcodebert/evaluation.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"eba6034b65efb5e6"}},{"code_sha256_prefix":"5f7f1f924be9f5a9","entry":"get_dnabert_encoder","repo":"bioscan-ml/BarcodeBERT","repo_kind":"official","path":"barcodebert/bzsl/models/model.py","file_url":"https://github.com/bioscan-ml/BarcodeBERT/blob/HEAD/barcodebert/bzsl/models/model.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"5f7f1f924be9f5a9"}},{"code_sha256_prefix":"7648cdeb15ca24f3","entry":"prepare","repo":"bioscan-ml/BarcodeBERT","repo_kind":"official","path":"barcodebert/utils.py","file_url":"https://github.com/bioscan-ml/BarcodeBERT/blob/HEAD/barcodebert/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"7648cdeb15ca24f3"}},{"code_sha256_prefix":"e30ded1705b3327c","entry":"remove_extra_pre_fix","repo":"bioscan-ml/BarcodeBERT","repo_kind":"official","path":"barcodebert/utils.py","file_url":"https://github.com/bioscan-ml/BarcodeBERT/blob/HEAD/barcodebert/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e30ded1705b3327c"}},{"code_sha256_prefix":"3aa2e236070ba9bd","entry":"representations_from_df","repo":"bioscan-ml/BarcodeBERT","repo_kind":"official","path":"barcodebert/datasets.py","file_url":"https://github.com/bioscan-ml/BarcodeBERT/blob/HEAD/barcodebert/datasets.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"3aa2e236070ba9bd"}},{"code_sha256_prefix":"58b9228a02f9d603","entry":"split_input_barcode_for_dnabert","repo":"bioscan-ml/BarcodeBERT","repo_kind":"official","path":"barcodebert/bzsl/models/model.py","file_url":"https://github.com/bioscan-ml/BarcodeBERT/blob/HEAD/barcodebert/bzsl/models/model.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"58b9228a02f9d603"}},{"code_sha256_prefix":"0af2290a64233a5e","entry":"train_one_epoch","repo":"bioscan-ml/BarcodeBERT","repo_kind":"official","path":"barcodebert/finetuning.py","file_url":"https://github.com/bioscan-ml/BarcodeBERT/blob/HEAD/barcodebert/finetuning.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"0af2290a64233a5e"}},{"code_sha256_prefix":"b3a9654b0d800dca","entry":"load_inference_model","repo":"bioscan-ml/BarcodeBERT","repo_kind":"official","path":"barcodebert/io.py","file_url":"https://github.com/bioscan-ml/BarcodeBERT/blob/HEAD/barcodebert/io.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b3a9654b0d800dca"}},{"code_sha256_prefix":"a3031f37064b5b3c","entry":"load_old_pretrained_model","repo":"bioscan-ml/BarcodeBERT","repo_kind":"official","path":"barcodebert/io.py","file_url":"https://github.com/bioscan-ml/BarcodeBERT/blob/HEAD/barcodebert/io.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a3031f37064b5b3c"}},{"code_sha256_prefix":"b3098f766bcf63bb","entry":"load_pretrained_model","repo":"bioscan-ml/BarcodeBERT","repo_kind":"official","path":"barcodebert/io.py","file_url":"https://github.com/bioscan-ml/BarcodeBERT/blob/HEAD/barcodebert/io.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b3098f766bcf63bb"}},{"code_sha256_prefix":"147fa47317265ee7","entry":"train_one_epoch","repo":"bioscan-ml/barcodebert","repo_kind":"official","path":"barcodebert/pretraining.py","file_url":"https://github.com/bioscan-ml/barcodebert/blob/HEAD/barcodebert/pretraining.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"147fa47317265ee7"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}