{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/machine-vision-therapy-multimodal-large","title":"Machine Vision Therapy: Multimodal Large Language Models Can Enhance Visual Robustness via Denoising In-Context Learning","arxiv_id":"2312.02546","date":"2023-12-05","proceeding":null,"authors":["Zhuo Huang","Chang Liu","Yinpeng Dong","Hang Su","Shibao Zheng","Tongliang Liu"],"abstract":"Although vision models such as Contrastive Language-Image Pre-Training (CLIP) show impressive generalization performance, their zero-shot robustness is still limited under Out-of-Distribution (OOD) scenarios without fine-tuning. Instead of undesirably providing human supervision as commonly done, it is possible to take advantage of Multi-modal Large Language Models (MLLMs) that hold powerful visual understanding abilities. However, MLLMs are shown to struggle with vision problems due to the incompatibility of tasks, thus hindering their utilization. In this paper, we propose to effectively leverage MLLMs to conduct Machine Vision Therapy which aims to rectify the noisy predictions from vision models. By fine-tuning with the denoised labels, the learning model performance can be boosted in an unsupervised manner. To solve the incompatibility issue, we propose a novel Denoising In-Context Learning (DICL) strategy to align vision tasks with MLLMs. Concretely, by estimating a transition matrix that captures the probability of one class being confused with another, an instruction containing a correct exemplar and an erroneous one from the most probable noisy class can be constructed. Such an instruction can help any MLLMs with ICL ability to detect and rectify incorrect predictions of vision models. Through extensive experiments on ImageNet, WILDS, DomainBed, and other OOD datasets, we carefully validate the quantitative and qualitative effectiveness of our method. Our code is available at https://github.com/tmllab/Machine_Vision_Therapy.","url_abs":"https://arxiv.org/abs/2312.02546v2","url_pdf":"https://arxiv.org/pdf/2312.02546v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"machine-vision-therapy-multimodal-large","repo_url":"https://github.com/tmllab/2024_icml_dicl","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"machine-vision-therapy-multimodal-large","repo_url":"https://github.com/tmllab/machine_vision_therapy","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"denoising","task_name":"Denoising"},{"task_slug":"in-context-learning","task_name":"In-Context Learning"}],"methods":[{"method_slug":"align","method_name":"ALIGN"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2312.02546","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2312.02546"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/tmllab/2024_icml_dicl","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/tmllab/machine_vision_therapy","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran_draft_wrong":1,"ran_fixture":1,"unverified":3},"by_repo_kind":{"official":{"samples":5,"ran":2,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"b1e9e0d3d7d7e615","entry":"maybe_dictionarize","repo":"tmllab/2024_icml_dicl","repo_kind":"official","path":"ImageNet_datasets/common.py","file_url":"https://github.com/tmllab/2024_icml_dicl/blob/HEAD/ImageNet_datasets/common.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b1e9e0d3d7d7e615"}},{"code_sha256_prefix":"9fbea808db47dc66","entry":"project_logits","repo":"tmllab/2024_icml_dicl","repo_kind":"official","path":"ImageNet_datasets/imagenet.py","file_url":"https://github.com/tmllab/2024_icml_dicl/blob/HEAD/ImageNet_datasets/imagenet.py","link_basis":"harvester_set","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"9fbea808db47dc66"}},{"code_sha256_prefix":"3903d266387afea6","entry":"get_classnames","repo":"tmllab/2024_icml_dicl","repo_kind":"official","path":"ImageNet_datasets/imagenet_classnames.py","file_url":"https://github.com/tmllab/2024_icml_dicl/blob/HEAD/ImageNet_datasets/imagenet_classnames.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"3903d266387afea6"}},{"code_sha256_prefix":"7727c964cfcd56e6","entry":"get_features","repo":"tmllab/2024_icml_dicl","repo_kind":"official","path":"ImageNet_datasets/common.py","file_url":"https://github.com/tmllab/2024_icml_dicl/blob/HEAD/ImageNet_datasets/common.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"7727c964cfcd56e6"}},{"code_sha256_prefix":"74edffa1dfc39e58","entry":"get_features_helper","repo":"tmllab/2024_icml_dicl","repo_kind":"official","path":"ImageNet_datasets/common.py","file_url":"https://github.com/tmllab/2024_icml_dicl/blob/HEAD/ImageNet_datasets/common.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"74edffa1dfc39e58"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}