{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/prototype-based-aleatoric-uncertainty-1","title":"Prototype-based Aleatoric Uncertainty Quantification for Cross-modal Retrieval","arxiv_id":"2309.17093","date":"2023-09-29","proceeding":"NeurIPS 2023 11","authors":["Hao Li","Jingkuan Song","Lianli Gao","Xiaosu Zhu","Heng Tao Shen"],"abstract":"Cross-modal Retrieval methods build similarity relations between vision and language modalities by jointly learning a common representation space. However, the predictions are often unreliable due to the Aleatoric uncertainty, which is induced by low-quality data, e.g., corrupt images, fast-paced videos, and non-detailed texts. In this paper, we propose a novel Prototype-based Aleatoric Uncertainty Quantification (PAU) framework to provide trustworthy predictions by quantifying the uncertainty arisen from the inherent data ambiguity. Concretely, we first construct a set of various learnable prototypes for each modality to represent the entire semantics subspace. Then Dempster-Shafer Theory and Subjective Logic Theory are utilized to build an evidential theoretical framework by associating evidence with Dirichlet Distribution parameters. The PAU model induces accurate uncertainty and reliable predictions for cross-modal retrieval. Extensive experiments are performed on four major benchmark datasets of MSR-VTT, MSVD, DiDeMo, and MS-COCO, demonstrating the effectiveness of our method. The code is accessible at https://github.com/leolee99/PAU.","url_abs":"https://arxiv.org/abs/2309.17093v3","url_pdf":"https://arxiv.org/pdf/2309.17093v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"prototype-based-aleatoric-uncertainty-1","repo_url":"https://github.com/leolee99/pau","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"cross-modal-retrieval","task_name":"Cross-Modal Retrieval"},{"task_slug":"image-text-matching","task_name":"Image-text matching"},{"task_slug":"image-to-text-retrieval","task_name":"Image-to-Text Retrieval"},{"task_slug":"retrieval","task_name":"Retrieval"},{"task_slug":"text-retrieval","task_name":"Text Retrieval"},{"task_slug":"text-to-video-retrieval","task_name":"Text to Video Retrieval"},{"task_slug":"uncertainty-quantification","task_name":"Uncertainty Quantification"},{"task_slug":"video-retrieval","task_name":"Video Retrieval"},{"task_slug":"video-to-text-retrieval","task_name":"Video to Text Retrieval"},{"task_slug":"video-text-retrieval","task_name":"Video-Text Retrieval"}],"methods":[{"method_slug":"pau","method_name":"PAU"}],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/video-retrieval-on-didemo","task":"Video Retrieval","dataset":"DiDeMo","model":"PAU","rank_in_archive_order":27,"of":40,"metrics":{"text-to-video Mean Rank":"12.9","text-to-video Median Rank":"2.0","text-to-video R@1":"48.6","text-to-video R@10":"84.5","text-to-video R@5":"76.0","video-to-text Mean Rank":"9.8","video-to-text Median Rank":"2.0","video-to-text R@1":"48.1","video-to-text R@10":"85.7","video-to-text R@5":"74.2"},"uses_additional_data":false},{"leaderboard":"/sota/video-retrieval-on-msr-vtt-1ka","task":"Video Retrieval","dataset":"MSR-VTT-1kA","model":"PAU","rank_in_archive_order":25,"of":63,"metrics":{"text-to-video Mean Rank":"14.0","text-to-video Median Rank":"2.0","text-to-video R@1":"48.5","text-to-video R@10":"82.5","text-to-video R@5":"72.7","video-to-text Mean Rank":"9.7","video-to-text Median Rank":"2.0","video-to-text R@1":"48.3","video-to-text R@10":"83.2","video-to-text R@5":"73.0"},"uses_additional_data":false},{"leaderboard":"/sota/video-retrieval-on-msvd","task":"Video Retrieval","dataset":"MSVD","model":"PAU","rank_in_archive_order":16,"of":24,"metrics":{"text-to-video Mean Rank":"9.6","text-to-video Median Rank":"2.0","text-to-video R@1":"47.3","text-to-video R@10":"85.5","text-to-video R@5":"77.4","video-to-text Mean Rank":"2.4","video-to-text Median Rank":"1.0","video-to-text R@1":"68.9","video-to-text R@10":"97.1","video-to-text R@5":"93.1"},"uses_additional_data":false}],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2309.17093","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2309.17093"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/leolee99/PAU","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/leolee99/pau","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran_draft_wrong":6,"ran_violates":1,"ran":2,"ran_honours":2,"ran_fixture":1,"unverified":7},"by_repo_kind":{"official":{"samples":19,"ran":12,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"0f786c407fb1ee4c","entry":"swish","repo":"leolee99/PAU","repo_kind":"official","path":"modules/until_module.py","file_url":"https://github.com/leolee99/PAU/blob/HEAD/modules/until_module.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":2,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"0f786c407fb1ee4c"}},{"code_sha256_prefix":"98f385d847636a3e","entry":"basic_clean","repo":"leolee99/PAU","repo_kind":"official","path":"modules/tokenization_clip.py","file_url":"https://github.com/leolee99/PAU/blob/HEAD/modules/tokenization_clip.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"98f385d847636a3e"}},{"code_sha256_prefix":"47c2715feba5fd6b","entry":"check_attr","repo":"leolee99/PAU","repo_kind":"official","path":"modules/modeling.py","file_url":"https://github.com/leolee99/PAU/blob/HEAD/modules/modeling.py","link_basis":"harvester_set","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"47c2715feba5fd6b"}},{"code_sha256_prefix":"f45b80d057798fed","entry":"compute_metrics","repo":"leolee99/PAU","repo_kind":"official","path":"metrics.py","file_url":"https://github.com/leolee99/PAU/blob/HEAD/metrics.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f45b80d057798fed"}},{"code_sha256_prefix":"50e1ffed03f484ec","entry":"gelu","repo":"leolee99/PAU","repo_kind":"official","path":"modules/until_module.py","file_url":"https://github.com/leolee99/PAU/blob/HEAD/modules/until_module.py","link_basis":"harvester_set","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"50e1ffed03f484ec"}},{"code_sha256_prefix":"d919ae32e5e4e616","entry":"get_pairs","repo":"leolee99/PAU","repo_kind":"official","path":"modules/tokenization_clip.py","file_url":"https://github.com/leolee99/PAU/blob/HEAD/modules/tokenization_clip.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d919ae32e5e4e616"}},{"code_sha256_prefix":"a0c23f10479a984e","entry":"init_device","repo":"leolee99/PAU","repo_kind":"official","path":"main_pau.py","file_url":"https://github.com/leolee99/PAU/blob/HEAD/main_pau.py","link_basis":"harvester_set","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a0c23f10479a984e"}},{"code_sha256_prefix":"66bf2e9b33bd27dd","entry":"tensor_text_to_video_metrics","repo":"leolee99/PAU","repo_kind":"official","path":"metrics.py","file_url":"https://github.com/leolee99/PAU/blob/HEAD/metrics.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"66bf2e9b33bd27dd"}},{"code_sha256_prefix":"d34a2226d2cf142e","entry":"update_attr","repo":"leolee99/PAU","repo_kind":"official","path":"modules/modeling.py","file_url":"https://github.com/leolee99/PAU/blob/HEAD/modules/modeling.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d34a2226d2cf142e"}},{"code_sha256_prefix":"af64ec220e8bcdbc","entry":"url_to_filename","repo":"leolee99/PAU","repo_kind":"official","path":"modules/file_utils.py","file_url":"https://github.com/leolee99/PAU/blob/HEAD/modules/file_utils.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"af64ec220e8bcdbc"}},{"code_sha256_prefix":"e0960afdb0d64aa7","entry":"warmup_linear","repo":"leolee99/PAU","repo_kind":"official","path":"modules/optimization.py","file_url":"https://github.com/leolee99/PAU/blob/HEAD/modules/optimization.py","link_basis":"harvester_set","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e0960afdb0d64aa7"}},{"code_sha256_prefix":"9542161e9640b858","entry":"whitespace_clean","repo":"leolee99/PAU","repo_kind":"official","path":"modules/tokenization_clip.py","file_url":"https://github.com/leolee99/PAU/blob/HEAD/modules/tokenization_clip.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"9542161e9640b858"}},{"code_sha256_prefix":"f5470c7056a406fe","entry":"build_model","repo":"leolee99/PAU","repo_kind":"official","path":"modules/module_clip.py","file_url":"https://github.com/leolee99/PAU/blob/HEAD/modules/module_clip.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f5470c7056a406fe"}},{"code_sha256_prefix":"349d780dd4a89a37","entry":"cached_path","repo":"leolee99/PAU","repo_kind":"official","path":"modules/file_utils.py","file_url":"https://github.com/leolee99/PAU/blob/HEAD/modules/file_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"349d780dd4a89a37"}},{"code_sha256_prefix":"db0ac56aaf6e35e6","entry":"filename_to_url","repo":"leolee99/PAU","repo_kind":"official","path":"modules/file_utils.py","file_url":"https://github.com/leolee99/PAU/blob/HEAD/modules/file_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"db0ac56aaf6e35e6"}},{"code_sha256_prefix":"32381787730f06b2","entry":"get_args","repo":"leolee99/PAU","repo_kind":"official","path":"main_pau.py","file_url":"https://github.com/leolee99/PAU/blob/HEAD/main_pau.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"32381787730f06b2"}},{"code_sha256_prefix":"b6f9f3930836fe3f","entry":"tensor_video_to_text_sim","repo":"leolee99/PAU","repo_kind":"official","path":"metrics.py","file_url":"https://github.com/leolee99/PAU/blob/HEAD/metrics.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b6f9f3930836fe3f"}},{"code_sha256_prefix":"59e4e730a9dc5dc6","entry":"warmup_constant","repo":"leolee99/PAU","repo_kind":"official","path":"modules/optimization.py","file_url":"https://github.com/leolee99/PAU/blob/HEAD/modules/optimization.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"59e4e730a9dc5dc6"}},{"code_sha256_prefix":"20b3d25ba1b796e5","entry":"warmup_cosine","repo":"leolee99/PAU","repo_kind":"official","path":"modules/optimization.py","file_url":"https://github.com/leolee99/PAU/blob/HEAD/modules/optimization.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"20b3d25ba1b796e5"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}