{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/fairclip-harnessing-fairness-in-vision","title":"FairCLIP: Harnessing Fairness in Vision-Language Learning","arxiv_id":"2403.19949","date":"2024-03-29","proceeding":"CVPR 2024 1","authors":["Yan Luo","Min Shi","Muhammad Osama Khan","Muhammad Muneeb Afzal","Hao Huang","Shuaihang Yuan","Yu Tian","Luo Song","Ava Kouhana","Tobias Elze","Yi Fang","Mengyu Wang"],"abstract":"Fairness is a critical concern in deep learning, especially in healthcare, where these models influence diagnoses and treatment decisions. Although fairness has been investigated in the vision-only domain, the fairness of medical vision-language (VL) models remains unexplored due to the scarcity of medical VL datasets for studying fairness. To bridge this research gap, we introduce the first fair vision-language medical dataset Harvard-FairVLMed that provides detailed demographic attributes, ground-truth labels, and clinical notes to facilitate an in-depth examination of fairness within VL foundation models. Using Harvard-FairVLMed, we conduct a comprehensive fairness analysis of two widely-used VL models (CLIP and BLIP2), pre-trained on both natural and medical domains, across four different protected attributes. Our results highlight significant biases in all VL models, with Asian, Male, Non-Hispanic, and Spanish being the preferred subgroups across the protected attributes of race, gender, ethnicity, and language, respectively. In order to alleviate these biases, we propose FairCLIP, an optimal-transport-based approach that achieves a favorable trade-off between performance and fairness by reducing the Sinkhorn distance between the overall sample distribution and the distributions corresponding to each demographic group. As the first VL dataset of its kind, Harvard-FairVLMed holds the potential to catalyze advancements in the development of machine learning models that are both ethically aware and clinically effective. Our dataset and code are available at https://ophai.hms.harvard.edu/datasets/harvard-fairvlmed10k.","url_abs":"https://arxiv.org/abs/2403.19949v2","url_pdf":"https://arxiv.org/pdf/2403.19949v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"fairclip-harnessing-fairness-in-vision","repo_url":"https://github.com/Harvard-Ophthalmology-AI-Lab/FairCLIP","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"fairclip-harnessing-fairness-in-vision","repo_url":"https://github.com/purdue-m2/robust_fairness_for_medical_image","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"fairness","task_name":"Fairness"}],"methods":[{"method_slug":"aware","method_name":"AWARE"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":null,"atlas_url":"https://app.syntology.ai/?focus=2403.19949","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2403.19949"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/purdue-m2/robust_fairness_for_medical_image","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/Harvard-Ophthalmology-AI-Lab/FairCLIP","reach":null}],"summary":{"ran":1,"unverified":3},"by_repo_kind":{"official":{"samples":1,"ran":1,"repositories":1},"listed":{"samples":3,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"c77ee92e8218db6e","entry":"Adversary_Net","repo":"Harvard-Ophthalmology-AI-Lab/FairCLIP","repo_kind":"official","path":"FairCLIP/src/modules.py","file_url":"https://github.com/Harvard-Ophthalmology-AI-Lab/FairCLIP/blob/HEAD/FairCLIP/src/modules.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c77ee92e8218db6e"}},{"code_sha256_prefix":"16b8bc0c4c1c1c87","entry":"compute_gradcam","repo":"purdue-m2/robust_fairness_for_medical_image","repo_kind":"listed","path":"LAVIS/lavis/models/blip_models/blip_image_text_matching.py","file_url":"https://github.com/purdue-m2/robust_fairness_for_medical_image/blob/HEAD/LAVIS/lavis/models/blip_models/blip_image_text_matching.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"16b8bc0c4c1c1c87"}},{"code_sha256_prefix":"3ead4b7caf1ee67e","entry":"truncate_note","repo":"purdue-m2/robust_fairness_for_medical_image","repo_kind":"listed","path":"mae/engine_finetune.py","file_url":"https://github.com/purdue-m2/robust_fairness_for_medical_image/blob/HEAD/mae/engine_finetune.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"3ead4b7caf1ee67e"}},{"code_sha256_prefix":"84d9d90b28891b49","entry":"weight_estimation","repo":"purdue-m2/robust_fairness_for_medical_image","repo_kind":"listed","path":"RobustFairCLIP/finetune_RobustFairCLIP.py","file_url":"https://github.com/purdue-m2/robust_fairness_for_medical_image/blob/HEAD/RobustFairCLIP/finetune_RobustFairCLIP.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"84d9d90b28891b49"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}