{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/css-contrastive-semantic-similarity-for","title":"CSS: Contrastive Semantic Similarity for Uncertainty Quantification of LLMs","arxiv_id":"2406.03158","date":"2024-06-05","proceeding":null,"authors":["Shuang Ao","Stefan Rueger","Advaith Siddharthan"],"abstract":"Despite the impressive capability of large language models (LLMs), knowing when to trust their generations remains an open challenge. The recent literature on uncertainty quantification of natural language generation (NLG) utilises a conventional natural language inference (NLI) classifier to measure the semantic dispersion of LLMs responses. These studies employ logits of NLI classifier for semantic clustering to estimate uncertainty. However, logits represent the probability of the predicted class and barely contain feature information for potential clustering. Alternatively, CLIP (Contrastive Language-Image Pre-training) performs impressively in extracting image-text pair features and measuring their similarity. To extend its usability, we propose Contrastive Semantic Similarity, the CLIP-based feature extraction module to obtain similarity features for measuring uncertainty for text pairs. We apply this method to selective NLG, which detects and rejects unreliable generations for better trustworthiness of LLMs. We conduct extensive experiments with three LLMs on several benchmark question-answering datasets with comprehensive evaluation metrics. Results show that our proposed method performs better in estimating reliable responses of LLMs than comparable baselines. Results show that our proposed method performs better in estimating reliable responses of LLMs than comparable baselines. The code are available at \\url{https://github.com/AoShuang92/css_uq_llms}.","url_abs":"https://arxiv.org/abs/2406.03158v1","url_pdf":"https://arxiv.org/pdf/2406.03158v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"css-contrastive-semantic-similarity-for","repo_url":"https://github.com/aoshuang92/css_uq_llms","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":"clustering","task_name":"Clustering"},{"task_slug":"natural-language-inference","task_name":"Natural Language Inference"},{"task_slug":"question-answering","task_name":"Question Answering"},{"task_slug":"semantic-similarity","task_name":"Semantic Similarity"},{"task_slug":"semantic-textual-similarity","task_name":"Semantic Textual Similarity"},{"task_slug":"text-generation","task_name":"Text Generation"},{"task_slug":"uncertainty-quantification","task_name":"Uncertainty Quantification"}],"methods":[{"method_slug":"clip","method_name":"CLIP"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2406.03158","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2406.03158"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/aoshuang92/css_uq_llms","reach":{"status":"ok"}}],"summary":{"ran_fixture":1,"ran":6,"unverified":4},"by_repo_kind":{"official":{"samples":11,"ran":7,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":11,"samples":[{"code_sha256_prefix":"b2800ffbe076300f","entry":"area_under_accuracy_coverage_curve","repo":"aoshuang92/css_uq_llms","repo_kind":"official","path":"UQ-CSS/pipeline/eval_uq.py","file_url":"https://github.com/aoshuang92/css_uq_llms/blob/HEAD/UQ-CSS/pipeline/eval_uq.py","link_basis":"plan_row","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"b2800ffbe076300f"}},{"code_sha256_prefix":"c36588258a092d01","entry":"get_affinity_mat","repo":"aoshuang92/css_uq_llms","repo_kind":"official","path":"UQ-CSS/pipeline/clustering.py","file_url":"https://github.com/aoshuang92/css_uq_llms/blob/HEAD/UQ-CSS/pipeline/clustering.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"c36588258a092d01"}},{"code_sha256_prefix":"e6fdbc463ce2485d","entry":"get_key_from_generated_strings_path_new","repo":"aoshuang92/css_uq_llms","repo_kind":"official","path":"UQ-CSS/dataeval/load.py","file_url":"https://github.com/aoshuang92/css_uq_llms/blob/HEAD/UQ-CSS/dataeval/load.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"e6fdbc463ce2485d"}},{"code_sha256_prefix":"55a1526dbd125e56","entry":"process_data_to_model_inputs","repo":"aoshuang92/css_uq_llms","repo_kind":"official","path":"UQ-CSS/dataeval/triviaqa.py","file_url":"https://github.com/aoshuang92/css_uq_llms/blob/HEAD/UQ-CSS/dataeval/triviaqa.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"55a1526dbd125e56"}},{"code_sha256_prefix":"f87b66b5cc6d6c6f","entry":"recover_sim_mat_new","repo":"aoshuang92/css_uq_llms","repo_kind":"official","path":"UQ-CSS/pipeline/uq_bb.py","file_url":"https://github.com/aoshuang92/css_uq_llms/blob/HEAD/UQ-CSS/pipeline/uq_bb.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"f87b66b5cc6d6c6f"}},{"code_sha256_prefix":"07b6dace596d7818","entry":"sample_to_prompt","repo":"aoshuang92/css_uq_llms","repo_kind":"official","path":"UQ-CSS/dataeval/nq_open.py","file_url":"https://github.com/aoshuang92/css_uq_llms/blob/HEAD/UQ-CSS/dataeval/nq_open.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"07b6dace596d7818"}},{"code_sha256_prefix":"0da895b25f424e3d","entry":"sample_to_prompt","repo":"aoshuang92/css_uq_llms","repo_kind":"official","path":"UQ-CSS/dataeval/triviaqa.py","file_url":"https://github.com/aoshuang92/css_uq_llms/blob/HEAD/UQ-CSS/dataeval/triviaqa.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"0da895b25f424e3d"}},{"code_sha256_prefix":"8d0c51ce26ff6e51","entry":"get_D_mat","repo":"aoshuang92/css_uq_llms","repo_kind":"official","path":"UQ-CSS/pipeline/clustering.py","file_url":"https://github.com/aoshuang92/css_uq_llms/blob/HEAD/UQ-CSS/pipeline/clustering.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"8d0c51ce26ff6e51"}},{"code_sha256_prefix":"cf36dbb016c0d562","entry":"get_L_mat","repo":"aoshuang92/css_uq_llms","repo_kind":"official","path":"UQ-CSS/pipeline/clustering.py","file_url":"https://github.com/aoshuang92/css_uq_llms/blob/HEAD/UQ-CSS/pipeline/clustering.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"cf36dbb016c0d562"}},{"code_sha256_prefix":"4046d4dc3282ea6c","entry":"get_dataset","repo":"aoshuang92/css_uq_llms","repo_kind":"official","path":"UQ-CSS/dataeval/nq_open.py","file_url":"https://github.com/aoshuang92/css_uq_llms/blob/HEAD/UQ-CSS/dataeval/nq_open.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"4046d4dc3282ea6c"}},{"code_sha256_prefix":"ded65a0007a6b079","entry":"get_dataset","repo":"aoshuang92/css_uq_llms","repo_kind":"official","path":"UQ-CSS/dataeval/triviaqa.py","file_url":"https://github.com/aoshuang92/css_uq_llms/blob/HEAD/UQ-CSS/dataeval/triviaqa.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"ded65a0007a6b079"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}