{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/evaluating-robustness-to-dataset-shift-via","title":"Evaluating Robustness to Dataset Shift via Parametric Robustness Sets","arxiv_id":"2205.15947","date":"2022-05-31","proceeding":null,"authors":["Nikolaj Thams","Michael Oberst","David Sontag"],"abstract":"We give a method for proactively identifying small, plausible shifts in distribution which lead to large differences in model performance. These shifts are defined via parametric changes in the causal mechanisms of observed variables, where constraints on parameters yield a \"robustness set\" of plausible distributions and a corresponding worst-case loss over the set. While the loss under an individual parametric shift can be estimated via reweighting techniques such as importance sampling, the resulting worst-case optimization problem is non-convex, and the estimate may suffer from large variance. For small shifts, however, we can construct a local second-order approximation to the loss under shift and cast the problem of finding a worst-case shift as a particular non-convex quadratic optimization problem, for which efficient algorithms are available. We demonstrate that this second-order approximation can be estimated directly for shifts in conditional exponential family models, and we bound the approximation error. We apply our approach to a computer vision task (classifying gender from images), revealing sensitivity to shifts in non-causal attributes.","url_abs":"https://arxiv.org/abs/2205.15947v4","url_pdf":"https://arxiv.org/pdf/2205.15947v4.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"evaluating-robustness-to-dataset-shift-via","repo_url":"https://github.com/clinicalml/parametric-robustness-evaluation","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"tf","reach":null}],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2205.15947","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2205.15947"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/clinicalml/parametric-robustness-evaluation","reach":null}],"summary":{"ran_draft_wrong":2,"ran_honours":1},"by_repo_kind":{"official":{"samples":3,"ran":3,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"d2ac18a7c86120fd","entry":"cond_mean_given_binary","repo":"clinicalml/parametric-robustness-evaluation","repo_kind":"official","path":"source/shift_gradients.py","file_url":"https://github.com/clinicalml/parametric-robustness-evaluation/blob/HEAD/source/shift_gradients.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d2ac18a7c86120fd"}},{"code_sha256_prefix":"ed2fe20a16afe225","entry":"convert_np_to_torch","repo":"clinicalml/parametric-robustness-evaluation","repo_kind":"official","path":"source/shift_gradients.py","file_url":"https://github.com/clinicalml/parametric-robustness-evaluation/blob/HEAD/source/shift_gradients.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ed2fe20a16afe225"}},{"code_sha256_prefix":"2050a4e77741d9b2","entry":"cov","repo":"clinicalml/parametric-robustness-evaluation","repo_kind":"official","path":"source/shift_gradients.py","file_url":"https://github.com/clinicalml/parametric-robustness-evaluation/blob/HEAD/source/shift_gradients.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"2050a4e77741d9b2"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}