{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/unifashion-a-unified-vision-language-model","title":"UniFashion: A Unified Vision-Language Model for Multimodal Fashion Retrieval and Generation","arxiv_id":"2408.11305","date":"2024-08-21","proceeding":null,"authors":["Xiangyu Zhao","Yuehan Zhang","Wenlong Zhang","Xiao-Ming Wu"],"abstract":"The fashion domain encompasses a variety of real-world multimodal tasks, including multimodal retrieval and multimodal generation. The rapid advancements in artificial intelligence generated content, particularly in technologies like large language models for text generation and diffusion models for visual generation, have sparked widespread research interest in applying these multimodal models in the fashion domain. However, tasks involving embeddings, such as image-to-text or text-to-image retrieval, have been largely overlooked from this perspective due to the diverse nature of the multimodal fashion domain. And current research on multi-task single models lack focus on image generation. In this work, we present UniFashion, a unified framework that simultaneously tackles the challenges of multimodal generation and retrieval tasks within the fashion domain, integrating image generation with retrieval tasks and text generation tasks. UniFashion unifies embedding and generative tasks by integrating a diffusion model and LLM, enabling controllable and high-fidelity generation. Our model significantly outperforms previous single-task state-of-the-art models across diverse fashion tasks, and can be readily adapted to manage complex vision-language tasks. This work demonstrates the potential learning synergy between multimodal generation and retrieval, offering a promising direction for future research in the fashion domain. The source code is available at https://github.com/xiangyu-mm/UniFashion.","url_abs":"https://arxiv.org/abs/2408.11305v2","url_pdf":"https://arxiv.org/pdf/2408.11305v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"unifashion-a-unified-vision-language-model","repo_url":"https://github.com/xiangyu-mm/unifashion","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":"image-generation","task_name":"Image Generation"},{"task_slug":"image-retrieval","task_name":"Image Retrieval"},{"task_slug":"image-to-text","task_name":"Image to text"},{"task_slug":"language-modeling","task_name":"Language Modeling"},{"task_slug":"language-modelling","task_name":"Language Modelling"},{"task_slug":"retrieval","task_name":"Retrieval"},{"task_slug":"text-generation","task_name":"Text Generation"},{"task_slug":"multimodal-generation","task_name":"multimodal generation"}],"methods":[{"method_slug":"diffusion","method_name":"Diffusion"},{"method_slug":"focus","method_name":"Focus"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2408.11305","atlas_url":"https://app.syntology.ai/?focus=2408.11305","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2408.11305"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/xiangyu-mm/UniFashion","reach":{"status":"ok"}}],"summary":{"ran_fixture":1,"ran_honours":1,"ran":5,"ran_violates":1,"unverified":3},"by_repo_kind":{"official":{"samples":11,"ran":8,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":11,"samples":[{"code_sha256_prefix":"f9ec7d70add02b6f","entry":"HWC3","repo":"xiangyu-mm/UniFashion","repo_kind":"official","path":"src/UNIStableVITON/dataset.py","file_url":"https://github.com/xiangyu-mm/UniFashion/blob/HEAD/src/UNIStableVITON/dataset.py","link_basis":"harvester_set","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"f9ec7d70add02b6f"}},{"code_sha256_prefix":"b6c1f837e6faaeba","entry":"collate_fn","repo":"xiangyu-mm/UniFashion","repo_kind":"official","path":"src/data_utils.py","file_url":"https://github.com/xiangyu-mm/UniFashion/blob/HEAD/src/data_utils.py","link_basis":"harvester_set","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"b6c1f837e6faaeba"}},{"code_sha256_prefix":"2d7caa5e94094b6d","entry":"generate_randomized_fiq_caption","repo":"xiangyu-mm/UniFashion","repo_kind":"official","path":"src/data_utils.py","file_url":"https://github.com/xiangyu-mm/UniFashion/blob/HEAD/src/data_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"2d7caa5e94094b6d"}},{"code_sha256_prefix":"e120c2f0865756b1","entry":"generate_randomized_fiq_caption","repo":"xiangyu-mm/UniFashion","repo_kind":"official","path":"src/predict.py","file_url":"https://github.com/xiangyu-mm/UniFashion/blob/HEAD/src/predict.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"e120c2f0865756b1"}},{"code_sha256_prefix":"0b909bb609d6fdce","entry":"imread","repo":"xiangyu-mm/UniFashion","repo_kind":"official","path":"src/UNIStableVITON/dataset.py","file_url":"https://github.com/xiangyu-mm/UniFashion/blob/HEAD/src/UNIStableVITON/dataset.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"0b909bb609d6fdce"}},{"code_sha256_prefix":"dfb6556db838ffdf","entry":"imread_for_albu","repo":"xiangyu-mm/UniFashion","repo_kind":"official","path":"src/UNIStableVITON/dataset.py","file_url":"https://github.com/xiangyu-mm/UniFashion/blob/HEAD/src/UNIStableVITON/dataset.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"dfb6556db838ffdf"}},{"code_sha256_prefix":"a1580dd1883758f6","entry":"squarepad_transform","repo":"xiangyu-mm/UniFashion","repo_kind":"official","path":"src/data_utils.py","file_url":"https://github.com/xiangyu-mm/UniFashion/blob/HEAD/src/data_utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"a1580dd1883758f6"}},{"code_sha256_prefix":"7c508037b40522af","entry":"str2bool","repo":"xiangyu-mm/UniFashion","repo_kind":"official","path":"src/cirr_test_submission.py","file_url":"https://github.com/xiangyu-mm/UniFashion/blob/HEAD/src/cirr_test_submission.py","link_basis":"harvester_set","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"7c508037b40522af"}},{"code_sha256_prefix":"55fe1b6faccdb503","entry":"get_peft_state_maybe_zero_3","repo":"xiangyu-mm/UniFashion","repo_kind":"official","path":"src/blip_fine_tune_2.py","file_url":"https://github.com/xiangyu-mm/UniFashion/blob/HEAD/src/blip_fine_tune_2.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"55fe1b6faccdb503"}},{"code_sha256_prefix":"b6d749eb9a3b3158","entry":"get_peft_state_non_lora_maybe_zero_3","repo":"xiangyu-mm/UniFashion","repo_kind":"official","path":"src/blip_fine_tune_2.py","file_url":"https://github.com/xiangyu-mm/UniFashion/blob/HEAD/src/blip_fine_tune_2.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"b6d749eb9a3b3158"}},{"code_sha256_prefix":"7890d7ff378dc2fc","entry":"maybe_zero_3","repo":"xiangyu-mm/UniFashion","repo_kind":"official","path":"src/blip_fine_tune_2.py","file_url":"https://github.com/xiangyu-mm/UniFashion/blob/HEAD/src/blip_fine_tune_2.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"7890d7ff378dc2fc"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}