{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/harnessing-the-power-of-mllms-for","title":"Harnessing the Power of MLLMs for Transferable Text-to-Image Person ReID","arxiv_id":"2405.04940","date":"2024-05-08","proceeding":"CVPR 2024 1","authors":["Wentao Tan","Changxing Ding","Jiayu Jiang","Fei Wang","Yibing Zhan","Dapeng Tao"],"abstract":"Text-to-image person re-identification (ReID) retrieves pedestrian images according to textual descriptions. Manually annotating textual descriptions is time-consuming, restricting the scale of existing datasets and therefore the generalization ability of ReID models. As a result, we study the transferable text-to-image ReID problem, where we train a model on our proposed large-scale database and directly deploy it to various datasets for evaluation. We obtain substantial training data via Multi-modal Large Language Models (MLLMs). Moreover, we identify and address two key challenges in utilizing the obtained textual descriptions. First, an MLLM tends to generate descriptions with similar structures, causing the model to overfit specific sentence patterns. Thus, we propose a novel method that uses MLLMs to caption images according to various templates. These templates are obtained using a multi-turn dialogue with a Large Language Model (LLM). Therefore, we can build a large-scale dataset with diverse textual descriptions. Second, an MLLM may produce incorrect descriptions. Hence, we introduce a novel method that automatically identifies words in a description that do not correspond with the image. This method is based on the similarity between one text and all patch token embeddings in the image. Then, we mask these words with a larger probability in the subsequent training epoch, alleviating the impact of noisy textual descriptions. The experimental results demonstrate that our methods significantly boost the direct transfer text-to-image ReID performance. Benefiting from the pre-trained model weights, we also achieve state-of-the-art performance in the traditional evaluation settings.","url_abs":"https://arxiv.org/abs/2405.04940v3","url_pdf":"https://arxiv.org/pdf/2405.04940v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"harnessing-the-power-of-mllms-for","repo_url":"https://github.com/wentaotan/mllm4text-reid","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":"language-modelling","task_name":"Language Modelling"},{"task_slug":"large-language-model","task_name":"Large Language Model"},{"task_slug":"person-re-identification","task_name":"Person Re-Identification"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2405.04940","atlas_url":"https://app.syntology.ai/?focus=2405.04940","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2405.04940"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/wentaotan/mllm4text-reid","reach":{"status":"ok"}}],"summary":{"ran":8,"ran_honours":1,"unverified":1},"by_repo_kind":{"official":{"samples":10,"ran":9,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":10,"samples":[{"code_sha256_prefix":"cb870e3cbf51d74a","entry":"build_transforms","repo":"wentaotan/mllm4text-reid","repo_kind":"official","path":"datasets/build.py","file_url":"https://github.com/wentaotan/mllm4text-reid/blob/HEAD/datasets/build.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"cb870e3cbf51d74a"}},{"code_sha256_prefix":"e1cb4f3d2779e452","entry":"collate","repo":"wentaotan/mllm4text-reid","repo_kind":"official","path":"datasets/build.py","file_url":"https://github.com/wentaotan/mllm4text-reid/blob/HEAD/datasets/build.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"e1cb4f3d2779e452"}},{"code_sha256_prefix":"d42185181d52b9d0","entry":"compute_patch","repo":"wentaotan/mllm4text-reid","repo_kind":"official","path":"model/objectives.py","file_url":"https://github.com/wentaotan/mllm4text-reid/blob/HEAD/model/objectives.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"d42185181d52b9d0"}},{"code_sha256_prefix":"17b7054641464ad3","entry":"find_mutual_nearest_neighbors","repo":"wentaotan/mllm4text-reid","repo_kind":"official","path":"model/objectives.py","file_url":"https://github.com/wentaotan/mllm4text-reid/blob/HEAD/model/objectives.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"17b7054641464ad3"}},{"code_sha256_prefix":"8fbaad777bcbec96","entry":"mc","repo":"wentaotan/mllm4text-reid","repo_kind":"official","path":"model/memory.py","file_url":"https://github.com/wentaotan/mllm4text-reid/blob/HEAD/model/memory.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"8fbaad777bcbec96"}},{"code_sha256_prefix":"0770b49b19f8c704","entry":"remove_punctuation_and_spaces","repo":"wentaotan/mllm4text-reid","repo_kind":"official","path":"datasets/cuhkpedes.py","file_url":"https://github.com/wentaotan/mllm4text-reid/blob/HEAD/datasets/cuhkpedes.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"0770b49b19f8c704"}},{"code_sha256_prefix":"489d084ef038d428","entry":"resize_pos_embed","repo":"wentaotan/mllm4text-reid","repo_kind":"official","path":"model/clip_model.py","file_url":"https://github.com/wentaotan/mllm4text-reid/blob/HEAD/model/clip_model.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"489d084ef038d428"}},{"code_sha256_prefix":"c0cf1196424d4f71","entry":"softmax","repo":"wentaotan/mllm4text-reid","repo_kind":"official","path":"datasets/bases.py","file_url":"https://github.com/wentaotan/mllm4text-reid/blob/HEAD/datasets/bases.py","link_basis":"harvester_set","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"c0cf1196424d4f71"}},{"code_sha256_prefix":"3d862712396854e3","entry":"tokenize","repo":"wentaotan/mllm4text-reid","repo_kind":"official","path":"datasets/bases.py","file_url":"https://github.com/wentaotan/mllm4text-reid/blob/HEAD/datasets/bases.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"3d862712396854e3"}},{"code_sha256_prefix":"ff8611f714fb9669","entry":"compute_part","repo":"wentaotan/mllm4text-reid","repo_kind":"official","path":"model/objectives.py","file_url":"https://github.com/wentaotan/mllm4text-reid/blob/HEAD/model/objectives.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"ff8611f714fb9669"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}