{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/cultureclip-empowering-clip-with-cultural","title":"CultureCLIP: Empowering CLIP with Cultural Awareness through Synthetic Images and Contextualized Captions","arxiv_id":"2507.06210","date":"2025-07-08","proceeding":null,"authors":["Yuchen Huang","Zhiyuan Fan","Zhitao He","Sandeep Polisetty","Wenyan Li","Yi R. Fung"],"abstract":"Pretrained vision-language models (VLMs) such as CLIP excel in multimodal understanding but struggle with contextually relevant fine-grained visual features, making it difficult to distinguish visually similar yet culturally distinct concepts. This limitation stems from the scarcity of high-quality culture-specific datasets, the lack of integrated contextual knowledge, and the absence of hard negatives highlighting subtle distinctions. To address these challenges, we first design a data curation pipeline that leverages open-sourced VLMs and text-to-image diffusion models to construct CulTwin, a synthetic cultural dataset. This dataset consists of paired concept-caption-image triplets, where concepts visually resemble each other but represent different cultural contexts. Then, we fine-tune CLIP on CulTwin to create CultureCLIP, which aligns cultural concepts with contextually enhanced captions and synthetic images through customized contrastive learning, enabling finer cultural differentiation while preserving generalization capabilities. Experiments on culturally relevant benchmarks show that CultureCLIP outperforms the base CLIP, achieving up to a notable 5.49% improvement in fine-grained concept recognition on certain tasks, while preserving CLIP's original generalization ability, validating the effectiveness of our data synthesis and VLM backbone training paradigm in capturing subtle cultural distinctions.","url_abs":"https://arxiv.org/abs/2507.06210v1","url_pdf":"https://arxiv.org/pdf/2507.06210v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"cultureclip-empowering-clip-with-cultural","repo_url":"https://github.com/lukahhcm/cultureclip","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":"contrastive-learning","task_name":"Contrastive Learning"}],"methods":[{"method_slug":"base","method_name":"BASE"},{"method_slug":"clip","method_name":"CLIP"},{"method_slug":"diffusion","method_name":"Diffusion"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":null,"atlas_url":"https://app.syntology.ai/?focus=2507.06210","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2507.06210"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/lukahhcm/cultureclip","reach":{"status":"ok"}}],"summary":{"unverified":11},"by_repo_kind":{"official":{"samples":11,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":11,"samples":[{"code_sha256_prefix":"8c5dfc37b403b890","entry":"L_caption_neg","repo":"lukahhcm/cultureclip","repo_kind":"official","path":"model_finetune/loss.py","file_url":"https://github.com/lukahhcm/cultureclip/blob/HEAD/model_finetune/loss.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"8c5dfc37b403b890"}},{"code_sha256_prefix":"98f3d9a2dbe966f8","entry":"clean_text","repo":"lukahhcm/cultureclip","repo_kind":"official","path":"data_curation/diverse_caption.py","file_url":"https://github.com/lukahhcm/cultureclip/blob/HEAD/data_curation/diverse_caption.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"98f3d9a2dbe966f8"}},{"code_sha256_prefix":"1d31de1c8207e447","entry":"clip_loss","repo":"lukahhcm/cultureclip","repo_kind":"official","path":"model_finetune/loss.py","file_url":"https://github.com/lukahhcm/cultureclip/blob/HEAD/model_finetune/loss.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"1d31de1c8207e447"}},{"code_sha256_prefix":"041e7ddf4cb7d30f","entry":"collate_fn","repo":"lukahhcm/cultureclip","repo_kind":"official","path":"model_finetune/data.py","file_url":"https://github.com/lukahhcm/cultureclip/blob/HEAD/model_finetune/data.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"041e7ddf4cb7d30f"}},{"code_sha256_prefix":"91cf5370b7ce3905","entry":"create_output","repo":"lukahhcm/cultureclip","repo_kind":"official","path":"data_curation/image_generation/image_gen.py","file_url":"https://github.com/lukahhcm/cultureclip/blob/HEAD/data_curation/image_generation/image_gen.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"91cf5370b7ce3905"}},{"code_sha256_prefix":"a960118eaac7f10f","entry":"extract_and_convert","repo":"lukahhcm/cultureclip","repo_kind":"official","path":"data_curation/bottom_up/filter.py","file_url":"https://github.com/lukahhcm/cultureclip/blob/HEAD/data_curation/bottom_up/filter.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"a960118eaac7f10f"}},{"code_sha256_prefix":"10112852bb5ceb00","entry":"get_dataloaders","repo":"lukahhcm/cultureclip","repo_kind":"official","path":"model_finetune/data.py","file_url":"https://github.com/lukahhcm/cultureclip/blob/HEAD/model_finetune/data.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"10112852bb5ceb00"}},{"code_sha256_prefix":"703d748ee2f6b2b3","entry":"load_data","repo":"lukahhcm/cultureclip","repo_kind":"official","path":"data_curation/image_generation/image_gen.py","file_url":"https://github.com/lukahhcm/cultureclip/blob/HEAD/data_curation/image_generation/image_gen.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"703d748ee2f6b2b3"}},{"code_sha256_prefix":"e138b1ea0b520a24","entry":"load_image_from_url","repo":"lukahhcm/cultureclip","repo_kind":"official","path":"data_curation/bottom_up/classification.py","file_url":"https://github.com/lukahhcm/cultureclip/blob/HEAD/data_curation/bottom_up/classification.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"e138b1ea0b520a24"}},{"code_sha256_prefix":"09ee60b87b3368f5","entry":"negclip_loss","repo":"lukahhcm/cultureclip","repo_kind":"official","path":"model_finetune/loss.py","file_url":"https://github.com/lukahhcm/cultureclip/blob/HEAD/model_finetune/loss.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"09ee60b87b3368f5"}},{"code_sha256_prefix":"4804b67d9ca85516","entry":"preprocess_image","repo":"lukahhcm/cultureclip","repo_kind":"official","path":"data_curation/bottom_up/classification.py","file_url":"https://github.com/lukahhcm/cultureclip/blob/HEAD/data_curation/bottom_up/classification.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"4804b67d9ca85516"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}