{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/making-the-most-of-text-semantics-to-improve","title":"Making the Most of Text Semantics to Improve Biomedical Vision--Language Processing","arxiv_id":"2204.09817","date":"2022-04-21","proceeding":null,"authors":["Benedikt Boecking","Naoto Usuyama","Shruthi Bannur","Daniel C. Castro","Anton Schwaighofer","Stephanie Hyland","Maria Wetscherek","Tristan Naumann","Aditya Nori","Javier Alvarez-Valle","Hoifung Poon","Ozan Oktay"],"abstract":"Multi-modal data abounds in biomedicine, such as radiology images and reports. Interpreting this data at scale is essential for improving clinical care and accelerating clinical research. Biomedical text with its complex semantics poses additional challenges in vision--language modelling compared to the general domain, and previous work has used insufficiently adapted models that lack domain-specific language understanding. In this paper, we show that principled textual semantic modelling can substantially improve contrastive learning in self-supervised vision--language processing. We release a language model that achieves state-of-the-art results in radiology natural language inference through its improved vocabulary and novel language pretraining objective leveraging semantics and discourse characteristics in radiology reports. Further, we propose a self-supervised joint vision--language approach with a focus on better text modelling. It establishes new state of the art results on a wide range of publicly available benchmarks, in part by leveraging our new domain-specific language model. We release a new dataset with locally-aligned phrase grounding annotations by radiologists to facilitate the study of complex semantic modelling in biomedical vision--language processing. A broad evaluation, including on this new dataset, shows that our contrastive learning approach, aided by textual-semantic modelling, outperforms prior methods in segmentation tasks, despite only using a global-alignment objective.","url_abs":"https://arxiv.org/abs/2204.09817v4","url_pdf":"https://arxiv.org/pdf/2204.09817v4.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"making-the-most-of-text-semantics-to-improve","repo_url":"https://github.com/microsoft/hi-ml/tree/main/hi-ml-multimodal","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"pytorch","reach":null},{"paper_slug":"making-the-most-of-text-semantics-to-improve","repo_url":"https://github.com/vios-s/ldm-phrase-grounding","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"contrastive-learning","task_name":"Contrastive Learning"},{"task_slug":"language-modeling","task_name":"Language Modeling"},{"task_slug":"language-modelling","task_name":"Language Modelling"},{"task_slug":"medical-image-classification","task_name":"Medical Image Classification"},{"task_slug":"natural-language-inference","task_name":"Natural Language Inference"},{"task_slug":"phrase-grounding","task_name":"Phrase Grounding"},{"task_slug":"pneumonia-detection","task_name":"Pneumonia Detection"}],"methods":[{"method_slug":"contrastive-learning","method_name":"Contrastive Learning"}],"datasets_introduced":[{"slug":"ms-cxr","name":"MS-CXR","full_name":"Making the Most of Text Semantics to Improve Biomedical Vision-Language Processing"}],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2204.09817","atlas_url":"https://app.syntology.ai/?focus=2204.09817","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2204.09817"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/microsoft/hi-ml/tree/main/hi-ml-multimodal","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/vios-s/ldm-phrase-grounding","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":3,"unverified":1},"by_repo_kind":{"official":{"samples":4,"ran":3,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"639ba99d63e88bd1","entry":"BertProjectionHead","repo":"microsoft/hi-ml","repo_kind":"official","path":"hi-ml-multimodal/src/health_multimodal/text/model/modelling_cxrbert.py","file_url":"https://github.com/microsoft/hi-ml/blob/HEAD/hi-ml-multimodal/src/health_multimodal/text/model/modelling_cxrbert.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"639ba99d63e88bd1"}},{"code_sha256_prefix":"aa375ca39177b49a","entry":"CXRBertConfig","repo":"microsoft/hi-ml","repo_kind":"official","path":"hi-ml-multimodal/src/health_multimodal/text/model/modelling_cxrbert.py","file_url":"https://github.com/microsoft/hi-ml/blob/HEAD/hi-ml-multimodal/src/health_multimodal/text/model/modelling_cxrbert.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"aa375ca39177b49a"}},{"code_sha256_prefix":"a00002413c3202bf","entry":"CXRBertOutput","repo":"microsoft/hi-ml","repo_kind":"official","path":"hi-ml-multimodal/src/health_multimodal/text/model/modelling_cxrbert.py","file_url":"https://github.com/microsoft/hi-ml/blob/HEAD/hi-ml-multimodal/src/health_multimodal/text/model/modelling_cxrbert.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a00002413c3202bf"}},{"code_sha256_prefix":"a219e03e037cfc36","entry":"CXRBertModel","repo":"microsoft/hi-ml","repo_kind":"official","path":"hi-ml-multimodal/src/health_multimodal/text/model/modelling_cxrbert.py","file_url":"https://github.com/microsoft/hi-ml/blob/HEAD/hi-ml-multimodal/src/health_multimodal/text/model/modelling_cxrbert.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a219e03e037cfc36"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}