{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/clap4clip-continual-learning-with","title":"CLAP4CLIP: Continual Learning with Probabilistic Finetuning for Vision-Language Models","arxiv_id":"2403.19137","date":"2024-03-28","proceeding":null,"authors":["Saurav Jha","Dong Gong","Lina Yao"],"abstract":"Continual learning (CL) aims to help deep neural networks learn new knowledge while retaining what has been learned. Owing to their powerful generalizability, pre-trained vision-language models such as Contrastive Language-Image Pre-training (CLIP) have lately gained traction as practical CL candidates. However, the domain mismatch between the pre-training and the downstream CL tasks often calls for finetuning of the CLIP on the latter. Most existing finetuning methods exhibit deterministic nature. This makes them overlook the many possible interactions across the input modalities and deems them unsafe for high-risk tasks requiring reliable uncertainty estimation. To address these, our work proposes Continual LeArning with Probabilistic finetuning (CLAP) - a probabilistic modeling framework over visual-guided text features per task, thus providing more calibrated CL finetuning. Unlike recent data-hungry anti-forgetting CL techniques, CLAP alleviates forgetting by exploiting the rich pre-trained knowledge of CLIP for weight initialization and distribution regularization of task-specific parameters. Cooperating with the diverse range of existing prompting methods, CLAP can surpass the predominant deterministic finetuning approaches for CL with CLIP. We conclude with out-of-the-box applications of superior uncertainty estimation abilities of CLAP including novel data detection and exemplar selection within the existing CL setups. Our code is available at \\url{https://github.com/srvCodes/clap4clip}.","url_abs":"https://arxiv.org/abs/2403.19137v3","url_pdf":"https://arxiv.org/pdf/2403.19137v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"clap4clip-continual-learning-with","repo_url":"https://github.com/srvcodes/clap4clip","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":"continual-learning","task_name":"Continual Learning"}],"methods":[{"method_slug":"clip","method_name":"CLIP"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2403.19137","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2403.19137"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/srvCodes/clap4clip","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/srvcodes/clap4clip","reach":{"status":"ok"}}],"summary":{"ran_draft_wrong":1,"ran":2,"unverified":2},"by_repo_kind":{"official":{"samples":5,"ran":3,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":5,"samples":[{"code_sha256_prefix":"024f44e235ffc943","entry":"ce_loss_np","repo":"srvCodes/clap4clip","repo_kind":"official","path":"classifier/utils.py","file_url":"https://github.com/srvCodes/clap4clip/blob/HEAD/classifier/utils.py","link_basis":"plan_row","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"024f44e235ffc943"}},{"code_sha256_prefix":"1cf46fbec279beb1","entry":"compute_uncertainty","repo":"srvCodes/clap4clip","repo_kind":"official","path":"classifier/utils.py","file_url":"https://github.com/srvCodes/clap4clip/blob/HEAD/classifier/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"1cf46fbec279beb1"}},{"code_sha256_prefix":"f2063c200be13daf","entry":"get_context_by_labels","repo":"srvCodes/clap4clip","repo_kind":"official","path":"classifier/utils.py","file_url":"https://github.com/srvCodes/clap4clip/blob/HEAD/classifier/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"f2063c200be13daf"}},{"code_sha256_prefix":"d2f7d254649d8fff","entry":"build_model","repo":"srvCodes/clap4clip","repo_kind":"official","path":"clip/model.py","file_url":"https://github.com/srvCodes/clap4clip/blob/HEAD/clip/model.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"d2f7d254649d8fff"}},{"code_sha256_prefix":"4e91c54386b18d34","entry":"get_backbone","repo":"srvCodes/clap4clip","repo_kind":"official","path":"utils/inc_net.py","file_url":"https://github.com/srvCodes/clap4clip/blob/HEAD/utils/inc_net.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"4e91c54386b18d34"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}