{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/diffusion-enhanced-test-time-adaptation-with","title":"Diffusion-Enhanced Test-time Adaptation with Text and Image Augmentation","arxiv_id":"2412.09706","date":"2024-12-12","proceeding":null,"authors":["Chun-Mei Feng","Yuanyang He","Jian Zou","Salman Khan","Huan Xiong","Zhen Li","WangMeng Zuo","Rick Siow Mong Goh","Yong liu"],"abstract":"Existing test-time prompt tuning (TPT) methods focus on single-modality data, primarily enhancing images and using confidence ratings to filter out inaccurate images. However, while image generation models can produce visually diverse images, single-modality data enhancement techniques still fail to capture the comprehensive knowledge provided by different modalities. Additionally, we note that the performance of TPT-based methods drops significantly when the number of augmented images is limited, which is not unusual given the computational expense of generative augmentation. To address these issues, we introduce IT3A, a novel test-time adaptation method that utilizes a pre-trained generative model for multi-modal augmentation of each test sample from unknown new domains. By combining augmented data from pre-trained vision and language models, we enhance the ability of the model to adapt to unknown new test data. Additionally, to ensure that key semantics are accurately retained when generating various visual and text enhancements, we employ cosine similarity filtering between the logits of the enhanced images and text with the original test data. This process allows us to filter out some spurious augmentation and inadequate combinations. To leverage the diverse enhancements provided by the generation model across different modals, we have replaced prompt tuning with an adapter for greater flexibility in utilizing text templates. Our experiments on the test datasets with distribution shifts and domain gaps show that in a zero-shot setting, IT3A outperforms state-of-the-art test-time prompt tuning methods with a 5.50% increase in accuracy.","url_abs":"https://arxiv.org/abs/2412.09706v2","url_pdf":"https://arxiv.org/pdf/2412.09706v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"diffusion-enhanced-test-time-adaptation-with","repo_url":"https://github.com/chunmeifeng/difftpt","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"image-augmentation","task_name":"Image Augmentation"},{"task_slug":"image-generation","task_name":"Image Generation"},{"task_slug":"test-time-adaptation","task_name":"Test-time Adaptation"}],"methods":[{"method_slug":"adapter","method_name":"Adapter"},{"method_slug":"focus","method_name":"Focus"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2412.09706","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2412.09706"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/chunmeifeng/difftpt","reach":null}],"summary":{"ran_draft_wrong":3,"ran_honours":1,"unverified":2},"by_repo_kind":{"official":{"samples":3,"ran":2,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":6,"samples":[{"code_sha256_prefix":"441ae80dd4f616f3","entry":"avg_entropy","repo":null,"repo_kind":null,"path":null,"file_url":null,"link_basis":"identical_code_first_harvested_elsewhere","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":null,"inline_ok":false,"mcp_get_code":{"code_sha256":"441ae80dd4f616f3"}},{"code_sha256_prefix":"f22e6ba381b580eb","entry":"avg_entropy","repo":"chunmeifeng/difftpt","repo_kind":"official","path":"IT3A/difftpt_utils.py","file_url":"https://github.com/chunmeifeng/difftpt/blob/HEAD/IT3A/difftpt_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"f22e6ba381b580eb"}},{"code_sha256_prefix":"3ce4d11e58de02ad","entry":"select_confident_samples","repo":null,"repo_kind":null,"path":null,"file_url":null,"link_basis":"identical_code_first_harvested_elsewhere","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":null,"inline_ok":false,"mcp_get_code":{"code_sha256":"3ce4d11e58de02ad"}},{"code_sha256_prefix":"69c3f2054e7f8df5","entry":"select_confident_samples_cosine","repo":"chunmeifeng/difftpt","repo_kind":"official","path":"IT3A/difftpt_utils.py","file_url":"https://github.com/chunmeifeng/difftpt/blob/HEAD/IT3A/difftpt_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"69c3f2054e7f8df5"}},{"code_sha256_prefix":"631c70c26c0fcedf","entry":"test_time_tuning","repo":null,"repo_kind":null,"path":null,"file_url":null,"link_basis":"identical_code_first_harvested_elsewhere","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":null,"inline_ok":false,"mcp_get_code":{"code_sha256":"631c70c26c0fcedf"}},{"code_sha256_prefix":"b4d461ec3c05c1b3","entry":"test_time_tuning_difftpt","repo":"chunmeifeng/difftpt","repo_kind":"official","path":"IT3A/difftpt_utils.py","file_url":"https://github.com/chunmeifeng/difftpt/blob/HEAD/IT3A/difftpt_utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"b4d461ec3c05c1b3"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}