{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/reprogramming-large-pretrained-language","title":"Reprogramming Pretrained Language Models for Antibody Sequence Infilling","arxiv_id":"2210.07144","date":"2022-10-05","proceeding":null,"authors":["Igor Melnyk","Vijil Chenthamarakshan","Pin-Yu Chen","Payel Das","Amit Dhurandhar","Inkit Padhi","Devleena Das"],"abstract":"Antibodies comprise the most versatile class of binding molecules, with numerous applications in biomedicine. Computational design of antibodies involves generating novel and diverse sequences, while maintaining structural consistency. Unique to antibodies, designing the complementarity-determining region (CDR), which determines the antigen binding affinity and specificity, creates its own unique challenges. Recent deep learning models have shown impressive results, however the limited number of known antibody sequence/structure pairs frequently leads to degraded performance, particularly lacking diversity in the generated sequences. In our work we address this challenge by leveraging Model Reprogramming (MR), which repurposes pretrained models on a source language to adapt to the tasks that are in a different language and have scarce data - where it may be difficult to train a high-performing model from scratch or effectively fine-tune an existing pre-trained model on the specific task. Specifically, we introduce ReprogBert in which a pretrained English language model is repurposed for protein sequence infilling - thus considers cross-language adaptation using less data. Results on antibody design benchmarks show that our model on low-resourced antibody sequence dataset provides highly diverse CDR sequences, up to more than a two-fold increase of diversity over the baselines, without losing structural integrity and naturalness. The generated sequences also demonstrate enhanced antigen binding specificity and virus neutralization ability. Code is available at https://github.com/IBM/ReprogBERT","url_abs":"https://arxiv.org/abs/2210.07144v2","url_pdf":"https://arxiv.org/pdf/2210.07144v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"reprogramming-large-pretrained-language","repo_url":"https://github.com/ibm/reprogbert","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"diversity","task_name":"Diversity"},{"task_slug":"language-modelling","task_name":"Language Modelling"},{"task_slug":"specificity","task_name":"Specificity"},{"task_slug":"text-infilling","task_name":"Text Infilling"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2210.07144","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2210.07144"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/ibm/reprogbert","reach":null}],"summary":{"ran":4,"unverified":3},"by_repo_kind":{"official":{"samples":7,"ran":4,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"a1a55d4e85621ec7","entry":"BertConfigProtein","repo":"ibm/reprogbert","repo_kind":"official","path":"models/ReprogBert.py","file_url":"https://github.com/ibm/reprogbert/blob/HEAD/models/ReprogBert.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"a1a55d4e85621ec7"}},{"code_sha256_prefix":"6ca397d85cca0690","entry":"ModifiedBertLMPredictionHead","repo":"ibm/reprogbert","repo_kind":"official","path":"models/ReprogBert.py","file_url":"https://github.com/ibm/reprogbert/blob/HEAD/models/ReprogBert.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"6ca397d85cca0690"}},{"code_sha256_prefix":"67dd34f3a2ef4690","entry":"ModifiedBertModel","repo":"ibm/reprogbert","repo_kind":"official","path":"models/ReprogBert.py","file_url":"https://github.com/ibm/reprogbert/blob/HEAD/models/ReprogBert.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"67dd34f3a2ef4690"}},{"code_sha256_prefix":"6b4c0868aa3aca0e","entry":"ModifiedBertPreTrainedModel","repo":"ibm/reprogbert","repo_kind":"official","path":"models/ReprogBert.py","file_url":"https://github.com/ibm/reprogbert/blob/HEAD/models/ReprogBert.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"6b4c0868aa3aca0e"}},{"code_sha256_prefix":"77372f4d71aefb84","entry":"BertForMaskedLMProt","repo":"ibm/reprogbert","repo_kind":"official","path":"models/ReprogBert.py","file_url":"https://github.com/ibm/reprogbert/blob/HEAD/models/ReprogBert.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"77372f4d71aefb84"}},{"code_sha256_prefix":"7d1f150806b0e41d","entry":"ModifiedBertEmbeddings","repo":"ibm/reprogbert","repo_kind":"official","path":"models/ReprogBert.py","file_url":"https://github.com/ibm/reprogbert/blob/HEAD/models/ReprogBert.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"7d1f150806b0e41d"}},{"code_sha256_prefix":"448244116f48091e","entry":"ModifiedBertOnlyMLMHead","repo":"ibm/reprogbert","repo_kind":"official","path":"models/ReprogBert.py","file_url":"https://github.com/ibm/reprogbert/blob/HEAD/models/ReprogBert.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"448244116f48091e"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}