{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/arxiv-2605-20490","title":"ECUAS$_n$: A family of metrics for principled evaluation of uncertainty-augmented systems","arxiv_id":"2605.20490","date":"2026-05-19","proceeding":null,"authors":["Lautaro Estienne","Erik Ernst","Matías Vera","Pablo Piantanida","Luciana Ferrer"],"abstract":"In high-stakes automated decision-making, access to predictive uncertainty is essential for enabling users -- human or downstream systems -- to accept or reject predictions based on application-specific cost trade-offs. Such uncertainty-augmented (UA) systems -- i.e., systems that output both predictions and uncertainty scores -- are currently being assessed in the literature in a variety of ways, using separate metrics to evaluate the predictions and the uncertainty scores, setting a cost function with a fixed rejection cost or integrating over a coverage-risk curve. We argue that these evaluation approaches are inadequate for assessing overall performance of the UA system for decision making under uncertainty and propose a novel family of metrics, ECUAS$_n$, formulated as proper scoring rules for the task of interest. The parameter $n$ controls the trade-off between the cost of incorrect predictions and imperfect uncertainties depending on the needs of the use-case. We demonstrate the advantages of the ECUAS$_n$ metrics both theoretically and empirically, through experiments on diverse classification and generation datasets, including a manually annotated subset of TriviaQA.","url_abs":"https://arxiv.org/abs/2605.20490","url_pdf":"https://arxiv.org/pdf/2605.20490","source":{"archive":null,"snapshot":"2025-07-28","note":"not in the Papers with Code archive (frozen at the snapshot)","row_kind":"graph","title_abstract_authors_date":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)"},"code_links":[],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":null,"mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2605.20490"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"mentioned_in_github":null,"is_official":null,"provenance":"deterministic:regex_extraction","mentioned_in_paper":null,"url":"https://github.com/erikernst4/callm","reach":null},{"mentioned_in_github":null,"is_official":null,"provenance":"deterministic:regex_extraction","mentioned_in_paper":null,"url":"https://github.com/luferrer/expected_cost","reach":null}],"summary":{"ran":4,"ran_draft_wrong":2,"ran_fixture":2,"ran_honours":2,"unverified":3},"by_repo_kind":{"found_in_text":{"samples":13,"ran":10,"repositories":2}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"818b04bf40101f71","entry":"CostMatrix","repo":"luferrer/expected_cost","repo_kind":"found_in_text","path":"expected_cost/ec.py","file_url":"https://github.com/luferrer/expected_cost/blob/HEAD/expected_cost/ec.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"818b04bf40101f71"}},{"code_sha256_prefix":"f853f957bc781793","entry":"answer_index_to_letter","repo":"erikernst4/callm","repo_kind":"found_in_text","path":"callm/prompts/mmlu.py","file_url":"https://github.com/erikernst4/callm/blob/HEAD/callm/prompts/mmlu.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"f853f957bc781793"}},{"code_sha256_prefix":"c7924f71b252f4be","entry":"average_cost_from_confusion_matrix","repo":"luferrer/expected_cost","repo_kind":"found_in_text","path":"expected_cost/ec.py","file_url":"https://github.com/luferrer/expected_cost/blob/HEAD/expected_cost/ec.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c7924f71b252f4be"}},{"code_sha256_prefix":"f0cc3213726c2e2f","entry":"bayes_decisions","repo":"luferrer/expected_cost","repo_kind":"found_in_text","path":"expected_cost/ec.py","file_url":"https://github.com/luferrer/expected_cost/blob/HEAD/expected_cost/ec.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f0cc3213726c2e2f"}},{"code_sha256_prefix":"2b843a69da666503","entry":"format_choices","repo":"erikernst4/callm","repo_kind":"found_in_text","path":"callm/prompts/mmlu.py","file_url":"https://github.com/erikernst4/callm/blob/HEAD/callm/prompts/mmlu.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"2b843a69da666503"}},{"code_sha256_prefix":"3e5b842949fde932","entry":"get_last_llm_outputs_path","repo":"erikernst4/callm","repo_kind":"found_in_text","path":"callm/utils.py","file_url":"https://github.com/erikernst4/callm/blob/HEAD/callm/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"3e5b842949fde932"}},{"code_sha256_prefix":"ca15dfbfeb8955ad","entry":"get_posteriors_from_scores","repo":"luferrer/expected_cost","repo_kind":"found_in_text","path":"expected_cost/ec.py","file_url":"https://github.com/luferrer/expected_cost/blob/HEAD/expected_cost/ec.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ca15dfbfeb8955ad"}},{"code_sha256_prefix":"3c440f21da67deeb","entry":"get_priors_from_data","repo":"luferrer/expected_cost","repo_kind":"found_in_text","path":"expected_cost/ec.py","file_url":"https://github.com/luferrer/expected_cost/blob/HEAD/expected_cost/ec.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"3c440f21da67deeb"}},{"code_sha256_prefix":"23f87553a731f93a","entry":"llks_to_logpost","repo":"luferrer/expected_cost","repo_kind":"found_in_text","path":"expected_cost/ec.py","file_url":"https://github.com/luferrer/expected_cost/blob/HEAD/expected_cost/ec.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"23f87553a731f93a"}},{"code_sha256_prefix":"8f4e6ee6e8b1f45b","entry":"llrs_to_logpost","repo":"luferrer/expected_cost","repo_kind":"found_in_text","path":"expected_cost/ec.py","file_url":"https://github.com/luferrer/expected_cost/blob/HEAD/expected_cost/ec.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"8f4e6ee6e8b1f45b"}},{"code_sha256_prefix":"cc74b9e05fc3002b","entry":"average_cost","repo":"luferrer/expected_cost","repo_kind":"found_in_text","path":"expected_cost/ec.py","file_url":"https://github.com/luferrer/expected_cost/blob/HEAD/expected_cost/ec.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"cc74b9e05fc3002b"}},{"code_sha256_prefix":"6d8e614bd1e91643","entry":"average_cost_for_bayes_decisions","repo":"luferrer/expected_cost","repo_kind":"found_in_text","path":"expected_cost/ec.py","file_url":"https://github.com/luferrer/expected_cost/blob/HEAD/expected_cost/ec.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"6d8e614bd1e91643"}},{"code_sha256_prefix":"596b735ec2604cb2","entry":"generalized_confusion_matrix","repo":"luferrer/expected_cost","repo_kind":"found_in_text","path":"expected_cost/ec.py","file_url":"https://github.com/luferrer/expected_cost/blob/HEAD/expected_cost/ec.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"596b735ec2604cb2"}}]},"arxiv_metadata":{"licence":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)","fields":["title","abstract","authors","date"],"primary_category":"cs.LG","source":"arxiv_2026.jsonl"},"syntology_extracted_results":null}