{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/learning-conformal-abstention-policies-for","title":"Learning Conformal Abstention Policies for Adaptive Risk Management in Large Language and Vision-Language Models","arxiv_id":"2502.06884","date":"2025-02-08","proceeding":null,"authors":["Sina Tayebati","Divake Kumar","Nastaran Darabi","Dinithi Jayasuriya","Ranganath Krishnan","Amit Ranjan Trivedi"],"abstract":"Large Language and Vision-Language Models (LLMs/VLMs) are increasingly used in safety-critical applications, yet their opaque decision-making complicates risk assessment and reliability. Uncertainty quantification (UQ) helps assess prediction confidence and enables abstention when uncertainty is high. Conformal prediction (CP), a leading UQ method, provides statistical guarantees but relies on static thresholds, which fail to adapt to task complexity and evolving data distributions, leading to suboptimal trade-offs in accuracy, coverage, and informativeness. To address this, we propose learnable conformal abstention, integrating reinforcement learning (RL) with CP to optimize abstention thresholds dynamically. By treating CP thresholds as adaptive actions, our approach balances multiple objectives, minimizing prediction set size while maintaining reliable coverage. Extensive evaluations across diverse LLM/VLM benchmarks show our method outperforms Least Ambiguous Classifiers (LAC) and Adaptive Prediction Sets (APS), improving accuracy by up to 3.2%, boosting AUROC for hallucination detection by 22.19%, enhancing uncertainty-guided selective generation (AUARC) by 21.17%, and reducing calibration error by 70%-85%. These improvements hold across multiple models and datasets while consistently meeting the 90% coverage target, establishing our approach as a more effective and flexible solution for reliable decision-making in safety-critical applications. The code is available at: {https://github.com/sinatayebati/vlm-uncertainty}.","url_abs":"https://arxiv.org/abs/2502.06884v1","url_pdf":"https://arxiv.org/pdf/2502.06884v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"learning-conformal-abstention-policies-for","repo_url":"https://github.com/sinatayebati/vlm-uncertainty","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"conformal-prediction","task_name":"Conformal Prediction"},{"task_slug":"decision-making","task_name":"Decision Making"},{"task_slug":"hallucination","task_name":"Hallucination"},{"task_slug":"informativeness","task_name":"Informativeness"},{"task_slug":"management","task_name":"Management"},{"task_slug":"prediction","task_name":"Prediction"},{"task_slug":"reinforcement-learning-1","task_name":"Reinforcement Learning (RL)"},{"task_slug":"uncertainty-quantification","task_name":"Uncertainty Quantification"}],"methods":[{"method_slug":"set","method_name":"SET"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2502.06884","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2502.06884"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/sinatayebati/vlm-uncertainty","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":6,"unverified":10},"by_repo_kind":{"official":{"samples":16,"ran":6,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"7ec8353674842469","entry":"get_options","repo":"sinatayebati/vlm-uncertainty","repo_kind":"official","path":"data_utils/common_utils.py","file_url":"https://github.com/sinatayebati/vlm-uncertainty/blob/HEAD/data_utils/common_utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"7ec8353674842469"}},{"code_sha256_prefix":"25593cfba3f7b483","entry":"is_none","repo":"sinatayebati/vlm-uncertainty","repo_kind":"official","path":"data_utils/common_utils.py","file_url":"https://github.com/sinatayebati/vlm-uncertainty/blob/HEAD/data_utils/common_utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"25593cfba3f7b483"}},{"code_sha256_prefix":"8a064d582b5c57ea","entry":"open_json","repo":"sinatayebati/vlm-uncertainty","repo_kind":"official","path":"data_utils/common_utils.py","file_url":"https://github.com/sinatayebati/vlm-uncertainty/blob/HEAD/data_utils/common_utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"8a064d582b5c57ea"}},{"code_sha256_prefix":"ca40ba6d177df427","entry":"process_ai2d","repo":"sinatayebati/vlm-uncertainty","repo_kind":"official","path":"data_utils/ai2d.py","file_url":"https://github.com/sinatayebati/vlm-uncertainty/blob/HEAD/data_utils/ai2d.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ca40ba6d177df427"}},{"code_sha256_prefix":"a1bac8b55fe3309a","entry":"process_seedbench","repo":"sinatayebati/vlm-uncertainty","repo_kind":"official","path":"data_utils/seedbench.py","file_url":"https://github.com/sinatayebati/vlm-uncertainty/blob/HEAD/data_utils/seedbench.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a1bac8b55fe3309a"}},{"code_sha256_prefix":"c1b3610ee2583172","entry":"process_single_sample","repo":"sinatayebati/vlm-uncertainty","repo_kind":"official","path":"data_utils/mmmu.py","file_url":"https://github.com/sinatayebati/vlm-uncertainty/blob/HEAD/data_utils/mmmu.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c1b3610ee2583172"}},{"code_sha256_prefix":"1267ef7cf3135661","entry":"download_ai2d","repo":"sinatayebati/vlm-uncertainty","repo_kind":"official","path":"data_utils/ai2d.py","file_url":"https://github.com/sinatayebati/vlm-uncertainty/blob/HEAD/data_utils/ai2d.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"1267ef7cf3135661"}},{"code_sha256_prefix":"a268ca9c33318e5c","entry":"download_mmbench","repo":"sinatayebati/vlm-uncertainty","repo_kind":"official","path":"data_utils/mmbench.py","file_url":"https://github.com/sinatayebati/vlm-uncertainty/blob/HEAD/data_utils/mmbench.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a268ca9c33318e5c"}},{"code_sha256_prefix":"25a29dad236f0992","entry":"download_seedbench","repo":"sinatayebati/vlm-uncertainty","repo_kind":"official","path":"data_utils/seedbench.py","file_url":"https://github.com/sinatayebati/vlm-uncertainty/blob/HEAD/data_utils/seedbench.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"25a29dad236f0992"}},{"code_sha256_prefix":"9e203cb8767712bd","entry":"get_dataset","repo":"sinatayebati/vlm-uncertainty","repo_kind":"official","path":"data_utils/utils.py","file_url":"https://github.com/sinatayebati/vlm-uncertainty/blob/HEAD/data_utils/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"9e203cb8767712bd"}},{"code_sha256_prefix":"4335b7a5c5a94aaf","entry":"get_logits","repo":"sinatayebati/vlm-uncertainty","repo_kind":"official","path":"models_utils/utils.py","file_url":"https://github.com/sinatayebati/vlm-uncertainty/blob/HEAD/models_utils/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"4335b7a5c5a94aaf"}},{"code_sha256_prefix":"2ef1e2c47c52a331","entry":"make_context","repo":"sinatayebati/vlm-uncertainty","repo_kind":"official","path":"input_utils/qwen_chat_utils.py","file_url":"https://github.com/sinatayebati/vlm-uncertainty/blob/HEAD/input_utils/qwen_chat_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"2ef1e2c47c52a331"}},{"code_sha256_prefix":"de4427887bed9253","entry":"parse_img_path","repo":"sinatayebati/vlm-uncertainty","repo_kind":"official","path":"data_utils/mmmu.py","file_url":"https://github.com/sinatayebati/vlm-uncertainty/blob/HEAD/data_utils/mmmu.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"de4427887bed9253"}},{"code_sha256_prefix":"eba2dfc02d4349ea","entry":"process_mmmu","repo":"sinatayebati/vlm-uncertainty","repo_kind":"official","path":"data_utils/mmmu.py","file_url":"https://github.com/sinatayebati/vlm-uncertainty/blob/HEAD/data_utils/mmmu.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"eba2dfc02d4349ea"}},{"code_sha256_prefix":"5c69a8ac5c608ef9","entry":"process_oodcv","repo":"sinatayebati/vlm-uncertainty","repo_kind":"official","path":"data_utils/oodcv.py","file_url":"https://github.com/sinatayebati/vlm-uncertainty/blob/HEAD/data_utils/oodcv.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"5c69a8ac5c608ef9"}},{"code_sha256_prefix":"d2a305808682d0b6","entry":"process_scienceqa","repo":"sinatayebati/vlm-uncertainty","repo_kind":"official","path":"data_utils/scienceqa.py","file_url":"https://github.com/sinatayebati/vlm-uncertainty/blob/HEAD/data_utils/scienceqa.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d2a305808682d0b6"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}