{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/medvoc-vocabulary-adaptation-for-fine-tuning","title":"MEDVOC: Vocabulary Adaptation for Fine-tuning Pre-trained Language Models on Medical Text Summarization","arxiv_id":"2405.04163","date":"2024-05-07","proceeding":null,"authors":["Gunjan Balde","Soumyadeep Roy","Mainack Mondal","Niloy Ganguly"],"abstract":"This work presents a dynamic vocabulary adaptation strategy, MEDVOC, for fine-tuning pre-trained language models (PLMs) like BertSumAbs, BART, and PEGASUS for improved medical text summarization. In contrast to existing domain adaptation approaches in summarization, MEDVOC treats vocabulary as an optimizable parameter and optimizes the PLM vocabulary based on fragment score conditioned only on the downstream task's reference summaries. Unlike previous works on vocabulary adaptation (limited only to classification tasks), optimizing vocabulary based on summarization tasks requires an extremely costly intermediate fine-tuning step on large summarization datasets. To that end, our novel fragment score-based hyperparameter search very significantly reduces this fine-tuning time -- from 450 days to less than 2 days on average. Furthermore, while previous works on vocabulary adaptation are often primarily tied to single PLMs, MEDVOC is designed to be deployable across multiple PLMs (with varying model vocabulary sizes, pre-training objectives, and model sizes) -- bridging the limited vocabulary overlap between the biomedical literature domain and PLMs. MEDVOC outperforms baselines by 15.74% in terms of Rouge-L in zero-shot setting and shows gains of 17.29% in high Out-Of-Vocabulary (OOV) concentrations. Our human evaluation shows MEDVOC generates more faithful medical summaries (88% compared to 59% in baselines). We make the codebase publicly available at https://github.com/gb-kgp/MEDVOC.","url_abs":"https://arxiv.org/abs/2405.04163v2","url_pdf":"https://arxiv.org/pdf/2405.04163v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"medvoc-vocabulary-adaptation-for-fine-tuning","repo_url":"https://github.com/gb-kgp/medvoc","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":"domain-adaptation","task_name":"Domain Adaptation"},{"task_slug":"text-summarization","task_name":"Text Summarization"}],"methods":[{"method_slug":"adam","method_name":"Adam"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"bart","method_name":"BART"},{"method_slug":"bpe","method_name":"BPE"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"pegasus","method_name":"PEGASUS"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"softmax","method_name":"Softmax"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2405.04163","atlas_url":"https://app.syntology.ai/?focus=2405.04163","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2405.04163"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/gb-kgp/medvoc","reach":{"status":"ok"}},{"provenance":"deterministic:regex_extraction","url":"https://github.com/gb-kgp/MEDVOC","reach":{"status":"ok"}}],"summary":{"ran":7,"ran_violates":1,"unverified":1},"by_repo_kind":{"official":{"samples":9,"ran":8,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":9,"samples":[{"code_sha256_prefix":"66575e439195e14f","entry":"all_gather_list","repo":"gb-kgp/MEDVOC","repo_kind":"official","path":"src/Model_Scripts/BertSumAbs_Modified_Scripts/src/distributed.py","file_url":"https://github.com/gb-kgp/MEDVOC/blob/HEAD/src/Model_Scripts/BertSumAbs_Modified_Scripts/src/distributed.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"66575e439195e14f"}},{"code_sha256_prefix":"7a7e9de09a2874c5","entry":"checkAllNum","repo":"gb-kgp/MEDVOC","repo_kind":"official","path":"src/Vocab_Scripts/GenerateCandidateWords/GenerateCandidateWords-BART.py","file_url":"https://github.com/gb-kgp/MEDVOC/blob/HEAD/src/Vocab_Scripts/GenerateCandidateWords/GenerateCandidateWords-BART.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"7a7e9de09a2874c5"}},{"code_sha256_prefix":"7b7669755eae20fe","entry":"has_repeat","repo":"gb-kgp/MEDVOC","repo_kind":"official","path":"src/Model_Scripts/BertSumAbs_Modified_Scripts/src/post_stats.py","file_url":"https://github.com/gb-kgp/MEDVOC/blob/HEAD/src/Model_Scripts/BertSumAbs_Modified_Scripts/src/post_stats.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"7b7669755eae20fe"}},{"code_sha256_prefix":"b80220994c939e73","entry":"is_master","repo":"gb-kgp/MEDVOC","repo_kind":"official","path":"src/Model_Scripts/BertSumAbs_Modified_Scripts/src/distributed.py","file_url":"https://github.com/gb-kgp/MEDVOC/blob/HEAD/src/Model_Scripts/BertSumAbs_Modified_Scripts/src/distributed.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"b80220994c939e73"}},{"code_sha256_prefix":"c3fcc4e906d8ecbd","entry":"n_grams","repo":"gb-kgp/MEDVOC","repo_kind":"official","path":"src/Model_Scripts/BertSumAbs_Modified_Scripts/src/post_stats.py","file_url":"https://github.com/gb-kgp/MEDVOC/blob/HEAD/src/Model_Scripts/BertSumAbs_Modified_Scripts/src/post_stats.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"c3fcc4e906d8ecbd"}},{"code_sha256_prefix":"11abfb4819fe2527","entry":"return_PAC_A","repo":"gb-kgp/MEDVOC","repo_kind":"official","path":"src/Vocab_Scripts/GenerateVocab-BART/GenerateSubwordsBART.py","file_url":"https://github.com/gb-kgp/MEDVOC/blob/HEAD/src/Vocab_Scripts/GenerateVocab-BART/GenerateSubwordsBART.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"11abfb4819fe2527"}},{"code_sha256_prefix":"14b39c4dc7a42e40","entry":"rouge_results_to_str","repo":"gb-kgp/MEDVOC","repo_kind":"official","path":"src/Model_Scripts/BertSumAbs_Modified_Scripts/src/cal_rouge.py","file_url":"https://github.com/gb-kgp/MEDVOC/blob/HEAD/src/Model_Scripts/BertSumAbs_Modified_Scripts/src/cal_rouge.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"14b39c4dc7a42e40"}},{"code_sha256_prefix":"f017532fc389cbfe","entry":"str2bool","repo":"gb-kgp/MEDVOC","repo_kind":"official","path":"src/Model_Scripts/BertSumAbs_Modified_Scripts/src/post_stats.py","file_url":"https://github.com/gb-kgp/MEDVOC/blob/HEAD/src/Model_Scripts/BertSumAbs_Modified_Scripts/src/post_stats.py","link_basis":"harvester_set","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"f017532fc389cbfe"}},{"code_sha256_prefix":"e3c19f394696636a","entry":"retain_common_set","repo":"gb-kgp/MEDVOC","repo_kind":"official","path":"src/Vocab_Scripts/GenerateVocab-BART/GenerateSubwordsBART.py","file_url":"https://github.com/gb-kgp/MEDVOC/blob/HEAD/src/Vocab_Scripts/GenerateVocab-BART/GenerateSubwordsBART.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"e3c19f394696636a"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}