{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/arxiv-2604-06854","title":"To Adapt or not to Adapt, Rethinking the Value of Medical Knowledge-Aware Large Language Models","arxiv_id":"2604.06854","date":"2026-04-08","proceeding":null,"authors":["Ane G. Domingo-Aldama","Iker De La Iglesia","Maitane Urruela","Aitziber Atutxa","Ander Barrena"],"abstract":"BACKGROUND: Recent studies have shown that domain-adapted large language models (LLMs) do not consistently outperform general-purpose counterparts on standard medical benchmarks, raising questions about the need for specialized clinical adaptation. METHODS: We systematically compare general and clinical LLMs on a diverse set of multiple choice clinical question answering tasks in English and Spanish. We introduce a perturbation based evaluation benchmark that probes model robustness, instruction following, and sensitivity to adversarial variations. Our evaluation includes, one-step and two-step question transformations, multi prompt testing and instruction guided assessment. We analyze a range of state-of-the-art clinical models and their general-purpose counterparts, focusing on Llama 3.1-based models. Additionally, we introduce Marmoka, a family of lightweight 8B-parameter clinical LLMs for English and Spanish, developed via continual domain-adaptive pretraining on medical corpora and instructions. RESULTS: The experiments show that clinical LLMs do not consistently outperform their general purpose counterparts on English clinical tasks, even under the proposed perturbation based benchmark. However, for the Spanish subsets the proposed Marmoka models obtain better results compared to Llama. CONCLUSIONS: Our results show that, under current short-form MCQA benchmarks, clinical LLMs offer only marginal and unstable improvements over general-purpose models in English, suggesting that existing evaluation frameworks may be insufficient to capture genuine medical expertise. We further find that both general and clinical models exhibit substantial limitations in instruction following and strict output formatting. Finally, we demonstrate that robust medical LLMs can be successfully developed for low-resource languages such as Spanish, as evidenced by the Marmoka models.","url_abs":"https://arxiv.org/abs/2604.06854","url_pdf":"https://arxiv.org/pdf/2604.06854","source":{"archive":null,"snapshot":"2025-07-28","note":"not in the Papers with Code archive (frozen at the snapshot)","row_kind":"graph","title_abstract_authors_date":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)"},"code_links":[],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2604.06854","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2604.06854"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"mentioned_in_github":null,"is_official":null,"provenance":"deterministic:regex_extraction","mentioned_in_paper":null,"url":"https://github.com/axolotl-ai-cloud/axolotl","reach":null},{"mentioned_in_github":null,"is_official":null,"provenance":"deterministic:regex_extraction","mentioned_in_paper":null,"url":"https://github.com/Maits27/Deduplication","reach":null}],"summary":{"unverified":7},"by_repo_kind":{"found_in_text":{"samples":7,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":7,"samples":[{"code_sha256_prefix":"e3af1bca4494043e","entry":"create_jsonl_dolma","repo":"Maits27/Deduplication","repo_kind":"found_in_text","path":"deduplication/dolma/preprocess/crear_jsonl_dolma.py","file_url":"https://github.com/Maits27/Deduplication/blob/HEAD/deduplication/dolma/preprocess/crear_jsonl_dolma.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"e3af1bca4494043e"}},{"code_sha256_prefix":"994f187be61e974b","entry":"create_jsonl_dolma","repo":"Maits27/Deduplication","repo_kind":"found_in_text","path":"deduplication/dolma/preprocess/crear_jsonl_txt_doble_enter.py","file_url":"https://github.com/Maits27/Deduplication/blob/HEAD/deduplication/dolma/preprocess/crear_jsonl_txt_doble_enter.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"994f187be61e974b"}},{"code_sha256_prefix":"b4ad859e77b335b0","entry":"eliminar_etiqueta_completa","repo":"Maits27/Deduplication","repo_kind":"found_in_text","path":"limpieza/limpiar.py","file_url":"https://github.com/Maits27/Deduplication/blob/HEAD/limpieza/limpiar.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"b4ad859e77b335b0"}},{"code_sha256_prefix":"0fe4337e533bcc08","entry":"limpiar_etiqueta_simple","repo":"Maits27/Deduplication","repo_kind":"found_in_text","path":"limpieza/limpiar.py","file_url":"https://github.com/Maits27/Deduplication/blob/HEAD/limpieza/limpiar.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"0fe4337e533bcc08"}},{"code_sha256_prefix":"7d8dcc211b3d2362","entry":"lineas_a_set","repo":"Maits27/Deduplication","repo_kind":"found_in_text","path":"wikipediaCrawler/eliminar_categorias_innecesarias.py","file_url":"https://github.com/Maits27/Deduplication/blob/HEAD/wikipediaCrawler/eliminar_categorias_innecesarias.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"7d8dcc211b3d2362"}},{"code_sha256_prefix":"366c7219adb664b0","entry":"read_txt_file","repo":"Maits27/Deduplication","repo_kind":"found_in_text","path":"deduplication/dolma/preprocess/crear_jsonl_de_txt_dolma.py","file_url":"https://github.com/Maits27/Deduplication/blob/HEAD/deduplication/dolma/preprocess/crear_jsonl_de_txt_dolma.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"366c7219adb664b0"}},{"code_sha256_prefix":"f20265615e665fbb","entry":"sanitize_text","repo":"Maits27/Deduplication","repo_kind":"found_in_text","path":"limpieza/limpiar.py","file_url":"https://github.com/Maits27/Deduplication/blob/HEAD/limpieza/limpiar.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"f20265615e665fbb"}}]},"arxiv_metadata":{"licence":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)","fields":["title","abstract","authors","date"],"primary_category":"cs.CL","source":"arxiv_2026.jsonl"},"syntology_extracted_results":null}