{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/post-hoc-probabilistic-vision-language-models","title":"Post-hoc Probabilistic Vision-Language Models","arxiv_id":"2412.06014","date":"2024-12-08","proceeding":null,"authors":["Anton Baumann","Rui Li","Marcus Klasson","Santeri Mentu","Shyamgopal Karthik","Zeynep Akata","Arno Solin","Martin Trapp"],"abstract":"Vision-language models (VLMs), such as CLIP and SigLIP, have found remarkable success in classification, retrieval, and generative tasks. For this, VLMs deterministically map images and text descriptions to a joint latent space in which their similarity is assessed using the cosine similarity. However, a deterministic mapping of inputs fails to capture uncertainties over concepts arising from domain shifts when used in downstream tasks. In this work, we propose post-hoc uncertainty estimation in VLMs that does not require additional training. Our method leverages a Bayesian posterior approximation over the last layers in VLMs and analytically quantifies uncertainties over cosine similarities. We demonstrate its effectiveness for uncertainty quantification and support set selection in active learning. Compared to baselines, we obtain improved and well-calibrated predictive uncertainties, interpretable uncertainty estimates, and sample-efficient active learning. Our results show promise for safety-critical applications of large-scale models.","url_abs":"https://arxiv.org/abs/2412.06014v1","url_pdf":"https://arxiv.org/pdf/2412.06014v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"post-hoc-probabilistic-vision-language-models","repo_url":"https://github.com/AaltoML/BayesVLM","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"active-learning","task_name":"Active Learning"},{"task_slug":"uncertainty-quantification","task_name":"Uncertainty Quantification"},{"task_slug":"zero-shot-image-classification","task_name":"Zero-Shot Image Classification"}],"methods":[{"method_slug":"clip","method_name":"CLIP"},{"method_slug":"set","method_name":"SET"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2412.06014","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2412.06014"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/naver-ai/prolip","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/AaltoML/BayesVLM","reach":null}],"summary":{"ran_honours":2,"ran_draft_wrong":5,"unverified":6},"by_repo_kind":{"official":{"samples":8,"ran":4,"repositories":1},"found_in_text":{"samples":5,"ran":3,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":5,"samples":[{"code_sha256_prefix":"84a0c40ffe322d12","entry":"_as_optional_tensor","repo":"AaltoML/BayesVLM","repo_kind":"official","path":"bayesvlm/hf/modeling_bayesvlm_clip.py","file_url":"https://github.com/AaltoML/BayesVLM/blob/HEAD/bayesvlm/hf/modeling_bayesvlm_clip.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"84a0c40ffe322d12"}},{"code_sha256_prefix":"c22a08de9aa51a86","entry":"_get_output","repo":"AaltoML/BayesVLM","repo_kind":"official","path":"bayesvlm/hf/modeling_bayesvlm_clip.py","file_url":"https://github.com/AaltoML/BayesVLM/blob/HEAD/bayesvlm/hf/modeling_bayesvlm_clip.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c22a08de9aa51a86"}},{"code_sha256_prefix":"b2ce6787aa1b9885","entry":"_normalize_mean_and_var","repo":"AaltoML/BayesVLM","repo_kind":"official","path":"bayesvlm/hf/modeling_bayesvlm_clip.py","file_url":"https://github.com/AaltoML/BayesVLM/blob/HEAD/bayesvlm/hf/modeling_bayesvlm_clip.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b2ce6787aa1b9885"}},{"code_sha256_prefix":"d07af93e1bec367e","entry":"_std_from_var","repo":"AaltoML/BayesVLM","repo_kind":"official","path":"bayesvlm/hf/modeling_bayesvlm_clip.py","file_url":"https://github.com/AaltoML/BayesVLM/blob/HEAD/bayesvlm/hf/modeling_bayesvlm_clip.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d07af93e1bec367e"}},{"code_sha256_prefix":"41856af97765c7ff","entry":"batched","repo":"naver-ai/prolip","repo_kind":"found_in_text","path":"src/prolip/zero_shot_classifier.py","file_url":"https://github.com/naver-ai/prolip/blob/HEAD/src/prolip/zero_shot_classifier.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"41856af97765c7ff"}},{"code_sha256_prefix":"4d038cd90610dfa0","entry":"get_input_dtype","repo":"naver-ai/prolip","repo_kind":"found_in_text","path":"src/prolip/model.py","file_url":"https://github.com/naver-ai/prolip/blob/HEAD/src/prolip/model.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"4d038cd90610dfa0"}},{"code_sha256_prefix":"feaac0348de86b5a","entry":"get_model_preprocess_cfg","repo":"naver-ai/prolip","repo_kind":"found_in_text","path":"src/prolip/model.py","file_url":"https://github.com/naver-ai/prolip/blob/HEAD/src/prolip/model.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"feaac0348de86b5a"}},{"code_sha256_prefix":"cf74575774d01c26","entry":"BayesVLMEmbeddingOutput","repo":"AaltoML/BayesVLM","repo_kind":"official","path":"bayesvlm/hf/modeling_bayesvlm_clip.py","file_url":"https://github.com/AaltoML/BayesVLM/blob/HEAD/bayesvlm/hf/modeling_bayesvlm_clip.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"cf74575774d01c26"}},{"code_sha256_prefix":"b4ce6a07560ec5ce","entry":"BayesVLMModel","repo":"AaltoML/BayesVLM","repo_kind":"official","path":"bayesvlm/hf/modeling_bayesvlm_clip.py","file_url":"https://github.com/AaltoML/BayesVLM/blob/HEAD/bayesvlm/hf/modeling_bayesvlm_clip.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b4ce6a07560ec5ce"}},{"code_sha256_prefix":"5aff72f64fe28e99","entry":"BayesVLMOutput","repo":"AaltoML/BayesVLM","repo_kind":"official","path":"bayesvlm/hf/modeling_bayesvlm_clip.py","file_url":"https://github.com/AaltoML/BayesVLM/blob/HEAD/bayesvlm/hf/modeling_bayesvlm_clip.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"5aff72f64fe28e99"}},{"code_sha256_prefix":"16e1a732a7be4602","entry":"_diag_cov","repo":"AaltoML/BayesVLM","repo_kind":"official","path":"bayesvlm/hf/modeling_bayesvlm_clip.py","file_url":"https://github.com/AaltoML/BayesVLM/blob/HEAD/bayesvlm/hf/modeling_bayesvlm_clip.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"16e1a732a7be4602"}},{"code_sha256_prefix":"90f3675f84d92e26","entry":"build_zero_shot_classifier_with_stds","repo":"naver-ai/prolip","repo_kind":"found_in_text","path":"src/prolip/zero_shot_classifier.py","file_url":"https://github.com/naver-ai/prolip/blob/HEAD/src/prolip/zero_shot_classifier.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"90f3675f84d92e26"}},{"code_sha256_prefix":"0529ec3d12d83d1c","entry":"get_cast_dtype","repo":"naver-ai/prolip","repo_kind":"found_in_text","path":"src/prolip/model.py","file_url":"https://github.com/naver-ai/prolip/blob/HEAD/src/prolip/model.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"0529ec3d12d83d1c"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}