Papers › Learning Under Treatment-Induced Label Indeterminacy with Expert Annotations of...

Learning Under Treatment-Induced Label Indeterminacy with Expert Annotations of Counterfactual Outcomes: A Case Study in Neurological Prognostication

12 Aug 2026arXiv:2608.12477added by Syntology

Xiaobin Shen, Chloe Y. H. Huang, Jonathan Elmer, George H. Chen

Title, abstract, authors and date from arXiv's metadata (CC0); this paper is not in the Papers with Code archive (frozen 2025-07-28).

Clinical prediction models are often developed as if the outcome of interest were cleanly observed for every patient. This assumption fails when treatment decisions make the clinically relevant outcome permanently unobservable. As a case study of this problem, we consider post-cardiac-arrest neurological prognostication using a cohort of 2,497 patients, including 1,429 patients whose outcomes were rendered indeterminate by treatment decisions. These patients with indeterminate outcomes were reviewed by independent clinical experts, who provided their guesses of counterfactual outcomes about what would have happened to the patients. We refer to these patients as uncertain cases. We also have patients for whom we observe their clinically relevant outcomes; we refer to these patients as certain cases. We propose a framework for evaluating prediction models that explicitly splits the evaluation between certain and uncertain cases. Here, we cannot easily evaluate both types of cases in a uniform manner as the available target labels differ. We then propose a simple prediction model that uses target labels from both certain and uncertain cases in a manner that allows us to trade off between them. Across the proposed neural model and a collection of tabular baselines, models with similar certain-case AUROC can nevertheless differ substantially in both certain-case Brier score and their probability estimates for uncertain cases. Improving alignment with target labels of uncertain cases for our proposed model generally comes at the cost of worse accuracy on certain cases, highlighting an explicit tradeoff that standard evaluation conceals. These results show that when treatment decisions determine whether clinically meaningful outcomes remain observable, conventional evaluation metrics can miss important failure modes in the very patients for whom prognostic support matters most.

Paper PDF Code Syntology ran

In Syntology Open this paper in Syntology's Atlas, the map of the papers in Syntology's graph and their citations.

For agents, Syntology's MCP tool lists every function and class Syntology harvested from this paper and whether it ran (how to connect): get_harvested_code_for_paper(arxiv_id="2608.12477")

Code

Syntology Ran 19 of 33 code samples harvested from 1 repository linked to this paper; 14 have no recorded run. Of those that ran: 2 ran · honoured contract; 11 ran · our draft was wrong; 2 ran · fixture could not drive it; 4 ran with no contract checked.

By repository: found in paper text by Syntology: 33 samples from 1 repository, 19 ran. The run record, sample by sample. “Ran” means executed on a synthesized input, not that the code is correct or reproduces the paper.

xiaobin-xs/learning-under-label-indeterminacy found in paper text by Syntology report

Reachability, where shown, is from one Syntology probe window (2026-09-16 to 2026-09-18); repositories not probed show nothing. GitHub stars are not tracked.

Code Syntology ran Syntology

33 samples harvested; 19 ran; 2 honoured the contract we drafted; 14 have no recorded run. Read from Syntology's graph 2026-09-24; that is when this build read the record, not when the samples ran.

2ran · honoured contract

11ran · our draft was wrong

2ran · fixture could not drive it

4ran

14unverified

Licence: 0 of the 33 samples are pointer only, meaning Syntology does not serve that copy's text. This page shows no code text for any sample; each one links to its file in the repository.

Harvested from xiaobin-xs/learning-under-label-indeterminacy. “Ran” means the sample executed on a synthesized input. It does not mean the output is correct, and nothing here reproduces the paper's results. “Honoured” and “violated” refer to a contract Syntology drafted from the code itself; “our draft was wrong” and “fixture could not drive it” are failures of Syntology's instrument, not of the code.

Each sample ends with its code_sha256, Syntology's identity for that exact code. An agent fetches the stored sample with Syntology's MCP tool get_code(code_sha256="…") (how to connect); click an identity to copy that call.

Repository labels, per sample. official repository: The archive marks this repository official for the paper. named in the paper: The archive records that the paper mentions this repository; it is not marked official. community (archive-listed): In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper. found in paper text by Syntology: Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted. community: Not in the archive's code links for this paper; a community repository Syntology harvested. Samples from a repository marked official are listed first. Licence labels name the repository's licence as recorded at harvest. “Pointer only” means Syntology does not serve that copy's text, for one of four reasons: no licence file was found; the licence was not identified; the licence is recorded as permissive but that copy's record is not marked cleared; or the licence is outside the permissive list Syntology serves text under (MIT, Apache-2.0, BSD and similar). Some licences outside that list permit redistribution, such as WTFPL, and GPL-3.0 under its conditions; they are simply not on the list. Hover a licence label for the reason. File links open the file on GitHub at the default branch, which may have changed since the harvest.

Dataset xiaobin-xs/learning-under-label-indeterminacy/src/evaluation/uncertain.py found in paper text by Syntology ran MIT (permissive) · 84af608b6dea72aa · report

ProbabilityInterval xiaobin-xs/learning-under-label-indeterminacy/src/evaluation/uncertain.py found in paper text by Syntology ran MIT (permissive) · 53835ebb58a07736 · report

UncertainEvaluationOutputs xiaobin-xs/learning-under-label-indeterminacy/src/evaluation/uncertain.py found in paper text by Syntology ran MIT (permissive) · 8663cc6061a0e317 · report

UncertaintySignalSpec xiaobin-xs/learning-under-label-indeterminacy/src/evaluation/uncertain.py found in paper text by Syntology ran MIT (permissive) · 34ea515b6dd4100f · report

_build_overall_summary xiaobin-xs/learning-under-label-indeterminacy/src/evaluation/uncertain.py found in paper text by Syntology ran · our draft was wrong MIT (permissive) · d3d4ccd299025786 · report

_resolve_baseline_family xiaobin-xs/learning-under-label-indeterminacy/src/evaluation/uncertain.py found in paper text by Syntology ran · our draft was wrong MIT (permissive) · bee5d063b70a432f · report

_safe_auprc xiaobin-xs/learning-under-label-indeterminacy/src/evaluation/uncertain.py found in paper text by Syntology ran · our draft was wrong MIT (permissive) · ccd178b7c7c7ad01 · report

_safe_auroc xiaobin-xs/learning-under-label-indeterminacy/src/evaluation/uncertain.py found in paper text by Syntology ran · our draft was wrong MIT (permissive) · 55a2b3235c7209b9 · report

_safe_fraction xiaobin-xs/learning-under-label-indeterminacy/src/evaluation/uncertain.py found in paper text by Syntology ran · honoured contract MIT (permissive) · 748c2dc270d62e67 · report

_safe_iqr xiaobin-xs/learning-under-label-indeterminacy/src/evaluation/uncertain.py found in paper text by Syntology ran · our draft was wrong MIT (permissive) · 10bcd380702646e6 · report

_safe_median xiaobin-xs/learning-under-label-indeterminacy/src/evaluation/uncertain.py found in paper text by Syntology ran · our draft was wrong MIT (permissive) · e4b30bbc6f7bc657 · report

_safe_spearman xiaobin-xs/learning-under-label-indeterminacy/src/evaluation/uncertain.py found in paper text by Syntology ran · fixture could not drive it MIT (permissive) · c024bf7c27da0bf4 · report

_signal_is_eligible_for_baseline xiaobin-xs/learning-under-label-indeterminacy/src/evaluation/uncertain.py found in paper text by Syntology ran · our draft was wrong MIT (permissive) · 5d9a8124c38aefa5 · report

_summarize_uncertain_subset xiaobin-xs/learning-under-label-indeterminacy/src/evaluation/uncertain.py found in paper text by Syntology ran · our draft was wrong MIT (permissive) · a4ffe51c1c43a222 · report

_top_fraction_conflict_rate xiaobin-xs/learning-under-label-indeterminacy/src/evaluation/uncertain.py found in paper text by Syntology ran · our draft was wrong MIT (permissive) · ec97d71bac3cf16a · report

_top_fraction_conflict_recall xiaobin-xs/learning-under-label-indeterminacy/src/evaluation/uncertain.py found in paper text by Syntology ran · our draft was wrong MIT (permissive) · e950b03f28761cf8 · report

aggregate_probabilities xiaobin-xs/learning-under-label-indeterminacy/src/evaluation/uncertain.py found in paper text by Syntology ran · fixture could not drive it MIT (permissive) · 4854c5722e6548bf · report

probabilities_to_disagreement_category xiaobin-xs/learning-under-label-indeterminacy/src/evaluation/uncertain.py found in paper text by Syntology ran · our draft was wrong MIT (permissive) · 80a4ed630e3ced17 · report

validate_score_6_cap xiaobin-xs/learning-under-label-indeterminacy/src/evaluation/uncertain.py found in paper text by Syntology ran · honoured contract fingerprinted MIT (permissive) · d7f2089b77f86f4e · report

_build_conflict_detection_summary xiaobin-xs/learning-under-label-indeterminacy/src/evaluation/uncertain.py found in paper text by Syntology unverified MIT (permissive) · b67989e5323c88cd · report

_build_group_summary xiaobin-xs/learning-under-label-indeterminacy/src/evaluation/uncertain.py found in paper text by Syntology unverified MIT (permissive) · 6a9b276b594d7c36 · report

_validate_uncertain_prediction_keys xiaobin-xs/learning-under-label-indeterminacy/src/evaluation/uncertain.py found in paper text by Syntology unverified MIT (permissive) · 77f48f2cf7fd5b00 · report

build_expert_reference_table xiaobin-xs/learning-under-label-indeterminacy/src/evaluation/uncertain.py found in paper text by Syntology unverified MIT (permissive) · dd02ac0a15cf727e · report

disagreement_category_to_group xiaobin-xs/learning-under-label-indeterminacy/src/evaluation/uncertain.py found in paper text by Syntology unverified MIT (permissive) · cd0e93748fdb396b · report

evaluate_uncertain_predictions xiaobin-xs/learning-under-label-indeterminacy/src/evaluation/uncertain.py found in paper text by Syntology unverified MIT (permissive) · 091c68ad2ebfd796 · report

get_probability_interval xiaobin-xs/learning-under-label-indeterminacy/src/evaluation/uncertain.py found in paper text by Syntology unverified MIT (permissive) · eaae070bd3d3fcb4 · report

score_is_on_or_above_1pct_side xiaobin-xs/learning-under-label-indeterminacy/src/evaluation/uncertain.py found in paper text by Syntology unverified MIT (permissive) · 3a72cd1f72a800d3 · report

score_to_probability xiaobin-xs/learning-under-label-indeterminacy/src/evaluation/uncertain.py found in paper text by Syntology unverified MIT (permissive) · d21ab5d69e26765c · report

scores_to_aggregated_probability xiaobin-xs/learning-under-label-indeterminacy/src/evaluation/uncertain.py found in paper text by Syntology unverified MIT (permissive) · 2d97efeb4334e823 · report

scores_to_disagreement_category xiaobin-xs/learning-under-label-indeterminacy/src/evaluation/uncertain.py found in paper text by Syntology unverified MIT (permissive) · 52420d1c582d9003 · report

scores_to_probabilities xiaobin-xs/learning-under-label-indeterminacy/src/evaluation/uncertain.py found in paper text by Syntology unverified MIT (permissive) · a099e7fbbde96d8b · report

scores_to_threshold_side_indicators xiaobin-xs/learning-under-label-indeterminacy/src/evaluation/uncertain.py found in paper text by Syntology unverified MIT (permissive) · b2b13d9da5ffb56d · report

scores_to_threshold_soft_target xiaobin-xs/learning-under-label-indeterminacy/src/evaluation/uncertain.py found in paper text by Syntology unverified MIT (permissive) · 5aef00475f9bf162 · report

Results from the paper

The Papers with Code archive ends with its 2025-07-28 snapshot. This paper's arXiv identifier, 2608.12477, was issued in August 2026, after that date, so the archive has no leaderboard rows for it.

Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections