{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/probvlm-probabilistic-adapter-for-frozen","title":"ProbVLM: Probabilistic Adapter for Frozen Vision-Language Models","arxiv_id":"2307.00398","date":"2023-07-01","proceeding":"ICCV 2023 1","authors":["Uddeshya Upadhyay","Shyamgopal Karthik","Massimiliano Mancini","Zeynep Akata"],"abstract":"Large-scale vision-language models (VLMs) like CLIP successfully find correspondences between images and text. Through the standard deterministic mapping process, an image or a text sample is mapped to a single vector in the embedding space. This is problematic: as multiple samples (images or text) can abstract the same concept in the physical world, deterministic embeddings do not reflect the inherent ambiguity in the embedding space. We propose ProbVLM, a probabilistic adapter that estimates probability distributions for the embeddings of pre-trained VLMs via inter/intra-modal alignment in a post-hoc manner without needing large-scale datasets or computing. On four challenging datasets, i.e., COCO, Flickr, CUB, and Oxford-flowers, we estimate the multi-modal embedding uncertainties for two VLMs, i.e., CLIP and BLIP, quantify the calibration of embedding uncertainties in retrieval tasks and show that ProbVLM outperforms other methods. Furthermore, we propose active learning and model selection as two real-world downstream tasks for VLMs and show that the estimated uncertainty aids both tasks. Lastly, we present a novel technique for visualizing the embedding distributions using a large-scale pre-trained latent diffusion model. Code is available at https://github.com/ExplainableML/ProbVLM.","url_abs":"https://arxiv.org/abs/2307.00398v3","url_pdf":"https://arxiv.org/pdf/2307.00398v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"probvlm-probabilistic-adapter-for-frozen","repo_url":"https://github.com/explainableml/probvlm","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"active-learning","task_name":"Active Learning"},{"task_slug":"model-selection","task_name":"Model Selection"},{"task_slug":"retrieval","task_name":"Retrieval"}],"methods":[{"method_slug":"adapter","method_name":"Adapter"},{"method_slug":"blip","method_name":"BLIP"},{"method_slug":"clip","method_name":"CLIP"},{"method_slug":"diffusion","method_name":"Diffusion"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2307.00398","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2307.00398"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/explainableml/probvlm","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran_draft_wrong":3,"unverified":12},"by_repo_kind":{"official":{"samples":15,"ran":3,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"98f385d847636a3e","entry":"basic_clean","repo":"explainableml/probvlm","repo_kind":"official","path":"src/clip/simple_tokenizer.py","file_url":"https://github.com/explainableml/probvlm/blob/HEAD/src/clip/simple_tokenizer.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"98f385d847636a3e"}},{"code_sha256_prefix":"d919ae32e5e4e616","entry":"get_pairs","repo":"explainableml/probvlm","repo_kind":"official","path":"src/clip/simple_tokenizer.py","file_url":"https://github.com/explainableml/probvlm/blob/HEAD/src/clip/simple_tokenizer.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d919ae32e5e4e616"}},{"code_sha256_prefix":"9542161e9640b858","entry":"whitespace_clean","repo":"explainableml/probvlm","repo_kind":"official","path":"src/clip/simple_tokenizer.py","file_url":"https://github.com/explainableml/probvlm/blob/HEAD/src/clip/simple_tokenizer.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"9542161e9640b858"}},{"code_sha256_prefix":"259eff225d89cc48","entry":"build_model","repo":"explainableml/probvlm","repo_kind":"official","path":"src/clip/model.py","file_url":"https://github.com/explainableml/probvlm/blob/HEAD/src/clip/model.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"259eff225d89cc48"}},{"code_sha256_prefix":"aee2e1c6decd89e9","entry":"caption_transform","repo":"explainableml/probvlm","repo_kind":"official","path":"src/ds/_transforms.py","file_url":"https://github.com/explainableml/probvlm/blob/HEAD/src/ds/_transforms.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"aee2e1c6decd89e9"}},{"code_sha256_prefix":"a37694f28caa41e1","entry":"eval_ProbVLM","repo":"explainableml/probvlm","repo_kind":"official","path":"src/train_probVLM.py","file_url":"https://github.com/explainableml/probvlm/blob/HEAD/src/train_probVLM.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a37694f28caa41e1"}},{"code_sha256_prefix":"9c748a328db870d6","entry":"eval_ProbVLM_HF","repo":"explainableml/probvlm","repo_kind":"official","path":"src/train_probVLM.py","file_url":"https://github.com/explainableml/probvlm/blob/HEAD/src/train_probVLM.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"9c748a328db870d6"}},{"code_sha256_prefix":"d621259cb2503f6f","entry":"get_pred_ranks","repo":"explainableml/probvlm","repo_kind":"official","path":"src/utils.py","file_url":"https://github.com/explainableml/probvlm/blob/HEAD/src/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d621259cb2503f6f"}},{"code_sha256_prefix":"9e65d0d94b1c84fe","entry":"get_recall","repo":"explainableml/probvlm","repo_kind":"official","path":"src/utils.py","file_url":"https://github.com/explainableml/probvlm/blob/HEAD/src/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"9e65d0d94b1c84fe"}},{"code_sha256_prefix":"cdb3b6f67b14988a","entry":"get_recall_COCOFLICKR","repo":"explainableml/probvlm","repo_kind":"official","path":"src/utils.py","file_url":"https://github.com/explainableml/probvlm/blob/HEAD/src/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"cdb3b6f67b14988a"}},{"code_sha256_prefix":"9e22cc211ad25fe0","entry":"image_to_caption_collate_fn","repo":"explainableml/probvlm","repo_kind":"official","path":"src/ds/_dataloader.py","file_url":"https://github.com/explainableml/probvlm/blob/HEAD/src/ds/_dataloader.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"9e22cc211ad25fe0"}},{"code_sha256_prefix":"d245fdedffda5d4f","entry":"imagenet_transform","repo":"explainableml/probvlm","repo_kind":"official","path":"src/ds/_transforms.py","file_url":"https://github.com/explainableml/probvlm/blob/HEAD/src/ds/_transforms.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d245fdedffda5d4f"}},{"code_sha256_prefix":"071ac72cc5321a2e","entry":"imagenet_transform_fn","repo":"explainableml/probvlm","repo_kind":"official","path":"src/ds/_dataloader.py","file_url":"https://github.com/explainableml/probvlm/blob/HEAD/src/ds/_dataloader.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"071ac72cc5321a2e"}},{"code_sha256_prefix":"9da6e7e7a1c0f8f6","entry":"load","repo":"explainableml/probvlm","repo_kind":"official","path":"src/clip/clip.py","file_url":"https://github.com/explainableml/probvlm/blob/HEAD/src/clip/clip.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"9da6e7e7a1c0f8f6"}},{"code_sha256_prefix":"841979de2ba28e2f","entry":"tokenize","repo":"explainableml/probvlm","repo_kind":"official","path":"src/ds/_transforms.py","file_url":"https://github.com/explainableml/probvlm/blob/HEAD/src/ds/_transforms.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"841979de2ba28e2f"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}