{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/watch-weighted-adaptive-testing-for","title":"WATCH: Adaptive Monitoring for AI Deployments via Weighted-Conformal Martingales","arxiv_id":"2505.04608","date":"2025-05-07","proceeding":null,"authors":["Drew Prinster","Xing Han","Anqi Liu","Suchi Saria"],"abstract":"Responsibly deploying artificial intelligence (AI) / machine learning (ML) systems in high-stakes settings arguably requires not only proof of system reliability, but moreover continual, post-deployment monitoring to quickly detect and address any unsafe behavior. Statistical methods for nonparametric change-point detection -- especially the tools of conformal test martingales (CTMs) and anytime-valid inference -- offer promising approaches to this monitoring task. However, existing methods are restricted to monitoring limited hypothesis classes or ``alarm criteria'' (such as data shifts that violate certain exchangeability assumptions), do not allow for online adaptation in response to shifts, and/or do not enable root-cause analysis of any degradation. In this paper, we expand the scope of these monitoring methods by proposing a weighted generalization of conformal test martingales (WCTMs), which lay a theoretical foundation for online monitoring for any unexpected changepoints in the data distribution while controlling false-alarms. For practical applications, we propose specific WCTM algorithms that adapt online to mild covariate shifts (in the marginal input distribution) while quickly detecting and diagnosing more severe shifts, such as concept shifts (in the conditional label distribution) or extreme (out-of-support) covariate shifts that cannot be easily adapted to. On real-world datasets, we demonstrate improved performance relative to state-of-the-art baselines.","url_abs":"https://arxiv.org/abs/2505.04608v2","url_pdf":"https://arxiv.org/pdf/2505.04608v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"watch-weighted-adaptive-testing-for","repo_url":"https://github.com/aaronhan223/wtr","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"change-point-detection","task_name":"Change Point Detection"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2505.04608","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2505.04608"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/aaronhan223/watch","reach":{"status":"ok","spdx":"BSD-3-Clause"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/aaronhan223/wtr","reach":null}],"summary":{"ran_violates":1,"ran_fixture":1,"unverified":22},"by_repo_kind":{"official":{"samples":3,"ran":2,"repositories":1},"found_in_text":{"samples":21,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"1036ba3a48cac196","entry":"calculate_p_values","repo":"aaronhan223/wtr","repo_kind":"official","path":"src/p_values.py","file_url":"https://github.com/aaronhan223/wtr/blob/HEAD/src/p_values.py","link_basis":"first_harvest_node","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"1036ba3a48cac196"}},{"code_sha256_prefix":"52fbf28f44e10714","entry":"calculate_p_values_and_quantiles","repo":"aaronhan223/wtr","repo_kind":"official","path":"src/p_values.py","file_url":"https://github.com/aaronhan223/wtr/blob/HEAD/src/p_values.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"52fbf28f44e10714"}},{"code_sha256_prefix":"24e257d88b978945","entry":"ResNet20","repo":"aaronhan223/watch","repo_kind":"found_in_text","path":"src/resnet.py","file_url":"https://github.com/aaronhan223/watch/blob/HEAD/src/resnet.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"24e257d88b978945"}},{"code_sha256_prefix":"d6c9d2e6d30cf13c","entry":"ResNet32","repo":"aaronhan223/watch","repo_kind":"found_in_text","path":"src/resnet.py","file_url":"https://github.com/aaronhan223/watch/blob/HEAD/src/resnet.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"d6c9d2e6d30cf13c"}},{"code_sha256_prefix":"3fe446beed24ab8f","entry":"brier_scores","repo":"aaronhan223/watch","repo_kind":"found_in_text","path":"src/podkopaev_ramdas/tests.py","file_url":"https://github.com/aaronhan223/watch/blob/HEAD/src/podkopaev_ramdas/tests.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"3fe446beed24ab8f"}},{"code_sha256_prefix":"5085978d4817f000","entry":"calculate_weighted_p_values_and_quantiles","repo":"aaronhan223/wtr","repo_kind":"official","path":"src/p_values.py","file_url":"https://github.com/aaronhan223/wtr/blob/HEAD/src/p_values.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"5085978d4817f000"}},{"code_sha256_prefix":"1e117d22b7948cfc","entry":"cusum_procedure","repo":"aaronhan223/watch","repo_kind":"found_in_text","path":"src/martingales.py","file_url":"https://github.com/aaronhan223/watch/blob/HEAD/src/martingales.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"1e117d22b7948cfc"}},{"code_sha256_prefix":"f7f73cf99f559888","entry":"effective_sample_size_est","repo":"aaronhan223/watch","repo_kind":"found_in_text","path":"src/utils.py","file_url":"https://github.com/aaronhan223/watch/blob/HEAD/src/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"f7f73cf99f559888"}},{"code_sha256_prefix":"8e67a8dac730d3c1","entry":"eval_loss_prob","repo":"aaronhan223/watch","repo_kind":"found_in_text","path":"src/main_mnist_cifar.py","file_url":"https://github.com/aaronhan223/watch/blob/HEAD/src/main_mnist_cifar.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"8e67a8dac730d3c1"}},{"code_sha256_prefix":"920241d05b739b86","entry":"evaluate","repo":"aaronhan223/watch","repo_kind":"found_in_text","path":"src/main_mnist_cifar.py","file_url":"https://github.com/aaronhan223/watch/blob/HEAD/src/main_mnist_cifar.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"920241d05b739b86"}},{"code_sha256_prefix":"3dcf309104240174","entry":"fit","repo":"aaronhan223/watch","repo_kind":"found_in_text","path":"src/main_pod_ram.py","file_url":"https://github.com/aaronhan223/watch/blob/HEAD/src/main_pod_ram.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"3dcf309104240174"}},{"code_sha256_prefix":"2a5d22e9de647d4d","entry":"get_meps_data","repo":"aaronhan223/watch","repo_kind":"found_in_text","path":"src/utils.py","file_url":"https://github.com/aaronhan223/watch/blob/HEAD/src/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"2a5d22e9de647d4d"}},{"code_sha256_prefix":"e3886065919c0b20","entry":"hoeffding_ci_lower_limit","repo":"aaronhan223/watch","repo_kind":"found_in_text","path":"src/podkopaev_ramdas/concentrations.py","file_url":"https://github.com/aaronhan223/watch/blob/HEAD/src/podkopaev_ramdas/concentrations.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"e3886065919c0b20"}},{"code_sha256_prefix":"bf57bb6d7f52d5f9","entry":"hoeffding_ci_upper_limit","repo":"aaronhan223/watch","repo_kind":"found_in_text","path":"src/podkopaev_ramdas/concentrations.py","file_url":"https://github.com/aaronhan223/watch/blob/HEAD/src/podkopaev_ramdas/concentrations.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"bf57bb6d7f52d5f9"}},{"code_sha256_prefix":"75e33f5c90046614","entry":"logistic_regression_weight_est","repo":"aaronhan223/watch","repo_kind":"found_in_text","path":"src/p_values.py","file_url":"https://github.com/aaronhan223/watch/blob/HEAD/src/p_values.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"75e33f5c90046614"}},{"code_sha256_prefix":"3616d89ef8f11c25","entry":"misclas_losses","repo":"aaronhan223/watch","repo_kind":"found_in_text","path":"src/podkopaev_ramdas/tests.py","file_url":"https://github.com/aaronhan223/watch/blob/HEAD/src/podkopaev_ramdas/tests.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"3616d89ef8f11c25"}},{"code_sha256_prefix":"4987abbabf98f992","entry":"phi_h","repo":"aaronhan223/watch","repo_kind":"found_in_text","path":"src/podkopaev_ramdas/concentrations.py","file_url":"https://github.com/aaronhan223/watch/blob/HEAD/src/podkopaev_ramdas/concentrations.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"4987abbabf98f992"}},{"code_sha256_prefix":"d5786e025273c2db","entry":"podkopaev_ramdas_algorithm1","repo":"aaronhan223/watch","repo_kind":"found_in_text","path":"src/podkopaev_ramdas/baseline_alg.py","file_url":"https://github.com/aaronhan223/watch/blob/HEAD/src/podkopaev_ramdas/baseline_alg.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"d5786e025273c2db"}},{"code_sha256_prefix":"c00e01fba06c1cf4","entry":"podkopaev_ramdas_changepoint_detection","repo":"aaronhan223/watch","repo_kind":"found_in_text","path":"src/podkopaev_ramdas/baseline_alg.py","file_url":"https://github.com/aaronhan223/watch/blob/HEAD/src/podkopaev_ramdas/baseline_alg.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"c00e01fba06c1cf4"}},{"code_sha256_prefix":"0a2eccd55a3473c8","entry":"random_forest_weight_est","repo":"aaronhan223/watch","repo_kind":"found_in_text","path":"src/p_values.py","file_url":"https://github.com/aaronhan223/watch/blob/HEAD/src/p_values.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"0a2eccd55a3473c8"}},{"code_sha256_prefix":"eee24789b6e59613","entry":"shiryaev_roberts_procedure","repo":"aaronhan223/watch","repo_kind":"found_in_text","path":"src/martingales.py","file_url":"https://github.com/aaronhan223/watch/blob/HEAD/src/martingales.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"eee24789b6e59613"}},{"code_sha256_prefix":"0eba4f72b27f3a5d","entry":"top_label_brier_scores","repo":"aaronhan223/watch","repo_kind":"found_in_text","path":"src/podkopaev_ramdas/tests.py","file_url":"https://github.com/aaronhan223/watch/blob/HEAD/src/podkopaev_ramdas/tests.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"0eba4f72b27f3a5d"}},{"code_sha256_prefix":"e224e4494d78ef17","entry":"train_one_epoch","repo":"aaronhan223/watch","repo_kind":"found_in_text","path":"src/main_mnist_cifar.py","file_url":"https://github.com/aaronhan223/watch/blob/HEAD/src/main_mnist_cifar.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"e224e4494d78ef17"}},{"code_sha256_prefix":"0c6a35aaa8280481","entry":"ville_procedure","repo":"aaronhan223/watch","repo_kind":"found_in_text","path":"src/martingales.py","file_url":"https://github.com/aaronhan223/watch/blob/HEAD/src/martingales.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"0c6a35aaa8280481"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}