{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/adapting-multi-modal-large-language-model-to","title":"Adapting Multi-modal Large Language Model to Concept Drift From Pre-training Onwards","arxiv_id":"2405.13459","date":"2024-05-22","proceeding":null,"authors":["Xiaoyu Yang","Jie Lu","En Yu"],"abstract":"Multi-modal Large Language Models (MLLMs) frequently face challenges from concept drift when dealing with real-world streaming data, wherein distributions change unpredictably. This mainly includes gradual drift due to long-tailed data and sudden drift from Out-Of-Distribution (OOD) data, both of which have increasingly drawn the attention of the research community. While these issues have been extensively studied in the individual domain of vision or language, their impacts on MLLMs in concept drift settings remain largely underexplored. In this paper, we reveal the susceptibility and vulnerability of Vision-Language (VL) models to significant biases arising from gradual drift and sudden drift, particularly in the pre-training. To effectively address these challenges, we propose a unified framework that extends concept drift theory to the multi-modal domain, enhancing the adaptability of the VL model to the distribution unpredictable changes. Additionally, a T-distribution based drift adapter is proposed to effectively mitigate the bias induced by the gradual drift, which also facilitates the model in distinguishing sudden distribution changes through explicit distribution modeling. Extensive experiments demonstrate our method enhances the efficiency and accuracy of image-text alignment in the pre-training of VL models, particularly in the concept drift scenario. Moreover, various downstream tasks exhibit significant improvements in our model's ability to adapt to long-tailed open world. Furthermore, we create a set of multi-modal datasets called OpenMMlo, specifically tailored for the long-tailed open world settings, to validate our findings. To foster the development of the multi-modal community, we have made both OpenMMlo datasets and our code publicly available at: https://github.com/Anonymous0Knight/ConceptDriftMLLMs.","url_abs":"https://arxiv.org/abs/2405.13459v2","url_pdf":"https://arxiv.org/pdf/2405.13459v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[],"tasks":[{"task_slug":"language-modeling","task_name":"Language Modeling"},{"task_slug":"language-modelling","task_name":"Language Modelling"},{"task_slug":"large-language-model","task_name":"Large Language Model"}],"methods":[{"method_slug":"adapter","method_name":"Adapter"},{"method_slug":"set","method_name":"SET"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2405.13459","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2405.13459"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/XiaoyuYoung/ConceptDriftMLLMs","reach":{"status":"ok","spdx":"BSD-3-Clause"}}],"summary":{"ran":7,"unverified":2},"by_repo_kind":{"found_in_text":{"samples":9,"ran":7,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"56f02812d66a0d11","entry":"generate_caption","repo":"XiaoyuYoung/ConceptDriftMLLMs","repo_kind":"found_in_text","path":"app/caption.py","file_url":"https://github.com/XiaoyuYoung/ConceptDriftMLLMs/blob/HEAD/app/caption.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"56f02812d66a0d11"}},{"code_sha256_prefix":"92f2e16ee0d3a24d","entry":"get_concat_v","repo":"XiaoyuYoung/ConceptDriftMLLMs","repo_kind":"found_in_text","path":"app/dataset_browser.py","file_url":"https://github.com/XiaoyuYoung/ConceptDriftMLLMs/blob/HEAD/app/dataset_browser.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"92f2e16ee0d3a24d"}},{"code_sha256_prefix":"13a43a7d815937e6","entry":"read_img","repo":"XiaoyuYoung/ConceptDriftMLLMs","repo_kind":"found_in_text","path":"app/calculate_coco_features.py","file_url":"https://github.com/XiaoyuYoung/ConceptDriftMLLMs/blob/HEAD/app/calculate_coco_features.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"13a43a7d815937e6"}},{"code_sha256_prefix":"926bb38f979a61ef","entry":"resize_img","repo":"XiaoyuYoung/ConceptDriftMLLMs","repo_kind":"found_in_text","path":"app/utils.py","file_url":"https://github.com/XiaoyuYoung/ConceptDriftMLLMs/blob/HEAD/app/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"926bb38f979a61ef"}},{"code_sha256_prefix":"f809f2ecf34c1c99","entry":"resize_img_w","repo":"XiaoyuYoung/ConceptDriftMLLMs","repo_kind":"found_in_text","path":"app/dataset_browser.py","file_url":"https://github.com/XiaoyuYoung/ConceptDriftMLLMs/blob/HEAD/app/dataset_browser.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"f809f2ecf34c1c99"}},{"code_sha256_prefix":"3d4bbb5ca53fd24a","entry":"sample_dataset","repo":"XiaoyuYoung/ConceptDriftMLLMs","repo_kind":"found_in_text","path":"app/dataset_browser.py","file_url":"https://github.com/XiaoyuYoung/ConceptDriftMLLMs/blob/HEAD/app/dataset_browser.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"3d4bbb5ca53fd24a"}},{"code_sha256_prefix":"cb33571427334815","entry":"tile","repo":"XiaoyuYoung/ConceptDriftMLLMs","repo_kind":"found_in_text","path":"lavis/models/base_model.py","file_url":"https://github.com/XiaoyuYoung/ConceptDriftMLLMs/blob/HEAD/lavis/models/base_model.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"cb33571427334815"}},{"code_sha256_prefix":"0ec9fc2025c16f65","entry":"all_gather_with_grad","repo":"XiaoyuYoung/ConceptDriftMLLMs","repo_kind":"found_in_text","path":"lavis/models/base_model.py","file_url":"https://github.com/XiaoyuYoung/ConceptDriftMLLMs/blob/HEAD/lavis/models/base_model.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"0ec9fc2025c16f65"}},{"code_sha256_prefix":"3e12ea8c09674aa8","entry":"compute_gradcam_batch","repo":"XiaoyuYoung/ConceptDriftMLLMs","repo_kind":"found_in_text","path":"app/multimodal_search.py","file_url":"https://github.com/XiaoyuYoung/ConceptDriftMLLMs/blob/HEAD/app/multimodal_search.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"3e12ea8c09674aa8"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}