{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/test-time-adaptation-with-clip-reward-for","title":"Test-Time Adaptation with CLIP Reward for Zero-Shot Generalization in Vision-Language Models","arxiv_id":"2305.18010","date":"2023-05-29","proceeding":null,"authors":["Shuai Zhao","Xiaohan Wang","Linchao Zhu","Yi Yang"],"abstract":"One fascinating aspect of pre-trained vision-language models~(VLMs) learning under language supervision is their impressive zero-shot generalization capability. However, this ability is hindered by distribution shifts between the training and testing data. Previous test time adaptation~(TTA) methods for VLMs in zero-shot classification rely on minimizing the entropy of model outputs, tending to be stuck in incorrect model predictions. In this work, we propose TTA with feedback to rectify the model output and prevent the model from becoming blindly confident. Specifically, a CLIP model is adopted as the reward model during TTA and provides feedback for the VLM. Given a single test sample, the VLM is forced to maximize the CLIP reward between the input and sampled results from the VLM output distribution. The proposed \\textit{reinforcement learning with CLIP feedback~(RLCF)} framework is highly flexible and universal. Beyond the classification task, with task-specific sampling strategies and a proper reward baseline choice, RLCF can be easily extended to not only discrimination tasks like retrieval but also generalization tasks like image captioning, improving the zero-shot generalization capacity of VLMs. According to the characteristics of these VL tasks, we build different fully TTA pipelines with RLCF to improve the zero-shot generalization ability of various VLMs. Extensive experiments along with promising empirical results demonstrate the effectiveness of RLCF. The code is available at https://github.com/mzhaoshuai/RLCF.","url_abs":"https://arxiv.org/abs/2305.18010v2","url_pdf":"https://arxiv.org/pdf/2305.18010v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"test-time-adaptation-with-clip-reward-for","repo_url":"https://github.com/mzhaoshuai/rlcf","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"image-captioning","task_name":"Image Captioning"},{"task_slug":"image-classification","task_name":"Image Classification"},{"task_slug":"retrieval","task_name":"Retrieval"},{"task_slug":"test-time-adaptation","task_name":"Test-time Adaptation"},{"task_slug":"text-retrieval","task_name":"Text Retrieval"},{"task_slug":"zero-shot-learning","task_name":"Zero-Shot Learning"},{"task_slug":"zero-shot-generalization","task_name":"Zero-shot Generalization"},{"task_slug":null,"task_name":"zero-shot-classification"}],"methods":[{"method_slug":"align","method_name":"ALIGN"},{"method_slug":"clip","method_name":"CLIP"},{"method_slug":"test","method_name":"Test"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2305.18010","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2305.18010"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/mzhaoshuai/RLCF","reach":{"status":"ok","spdx":"Apache-2.0"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/mzhaoshuai/rlcf","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran_draft_wrong":2,"ran":1,"unverified":7},"by_repo_kind":{"official":{"samples":10,"ran":3,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"441ae80dd4f616f3","entry":"avg_entropy","repo":"mzhaoshuai/RLCF","repo_kind":"official","path":"TPT/tpt_cls.py","file_url":"https://github.com/mzhaoshuai/RLCF/blob/HEAD/TPT/tpt_cls.py","link_basis":"plan_row","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"441ae80dd4f616f3"}},{"code_sha256_prefix":"d14ec1d960b2878e","entry":"extract_all_images","repo":"mzhaoshuai/RLCF","repo_kind":"official","path":"clipscore/clipscore.py","file_url":"https://github.com/mzhaoshuai/RLCF/blob/HEAD/clipscore/clipscore.py","link_basis":"plan_row","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"d14ec1d960b2878e"}},{"code_sha256_prefix":"3ce4d11e58de02ad","entry":"select_confident_samples","repo":"mzhaoshuai/RLCF","repo_kind":"official","path":"TPT/tpt_cls.py","file_url":"https://github.com/mzhaoshuai/RLCF/blob/HEAD/TPT/tpt_cls.py","link_basis":"plan_row","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"3ce4d11e58de02ad"}},{"code_sha256_prefix":"81d4b057b05c4ac1","entry":"avg_entropy","repo":"mzhaoshuai/RLCF","repo_kind":"official","path":"TPT/tpt_cls_rl.py","file_url":"https://github.com/mzhaoshuai/RLCF/blob/HEAD/TPT/tpt_cls_rl.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"81d4b057b05c4ac1"}},{"code_sha256_prefix":"aa56b568a90f8516","entry":"build_model","repo":"mzhaoshuai/RLCF","repo_kind":"official","path":"TPT/clip/model.py","file_url":"https://github.com/mzhaoshuai/RLCF/blob/HEAD/TPT/clip/model.py","link_basis":"plan_row","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"aa56b568a90f8516"}},{"code_sha256_prefix":"6a0c8049db0c43ed","entry":"get_args","repo":"mzhaoshuai/RLCF","repo_kind":"official","path":"TPT/params.py","file_url":"https://github.com/mzhaoshuai/RLCF/blob/HEAD/TPT/params.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"6a0c8049db0c43ed"}},{"code_sha256_prefix":"2fc6fdb85f13dc41","entry":"none_or_str","repo":"mzhaoshuai/RLCF","repo_kind":"official","path":"TPT/params.py","file_url":"https://github.com/mzhaoshuai/RLCF/blob/HEAD/TPT/params.py","link_basis":"plan_row","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"2fc6fdb85f13dc41"}},{"code_sha256_prefix":"03602a2e9952cb27","entry":"select_confident_samples","repo":"mzhaoshuai/RLCF","repo_kind":"official","path":"TPT/tpt_cls_rl.py","file_url":"https://github.com/mzhaoshuai/RLCF/blob/HEAD/TPT/tpt_cls_rl.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"03602a2e9952cb27"}},{"code_sha256_prefix":"5a1f70712c09b44b","entry":"test_time_tuning","repo":"mzhaoshuai/rlcf","repo_kind":"official","path":"TPT/tpt_cls_rl.py","file_url":"https://github.com/mzhaoshuai/rlcf/blob/HEAD/TPT/tpt_cls_rl.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"5a1f70712c09b44b"}},{"code_sha256_prefix":"2322616e6a60f9c6","entry":"test_time_tuning","repo":"mzhaoshuai/RLCF","repo_kind":"official","path":"TPT/tpt_cls.py","file_url":"https://github.com/mzhaoshuai/RLCF/blob/HEAD/TPT/tpt_cls.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"2322616e6a60f9c6"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}