{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/statistical-testing-under-distributional","title":"Statistical Testing under Distributional Shifts","arxiv_id":"2105.10821","date":"2021-05-22","proceeding":null,"authors":["Nikolaj Thams","Sorawit Saengkyongam","Niklas Pfister","Jonas Peters"],"abstract":"In this work, we introduce statistical testing under distributional shifts. We are interested in the hypothesis $P^* \\in H_0$ for a target distribution $P^*$, but observe data from a different distribution $Q^*$. We assume that $P^*$ is related to $Q^*$ through a known shift $\\tau$ and formally introduce hypothesis testing in this setting. We propose a general testing procedure that first resamples from the observed data to construct an auxiliary data set and then applies an existing test in the target domain. We prove that if the size of the resample is at most $o(\\sqrt{n})$ and the resampling weights are well-behaved, this procedure inherits the pointwise asymptotic level and power from the target test. If the map $\\tau$ is estimated from data, we can maintain the above guarantees under mild conditions if the estimation works sufficiently well. We further extend our results to finite sample level, uniform asymptotic level and a different resampling scheme. Testing under distributional shifts allows us to tackle a diverse set of problems. We argue that it may prove useful in reinforcement learning and covariate shift, we show how it reduces conditional to unconditional independence testing and we provide example applications in causal inference.","url_abs":"https://arxiv.org/abs/2105.10821v4","url_pdf":"https://arxiv.org/pdf/2105.10821v4.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"links_only","authors_date_abstract":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license), from the Kaggle arXiv metadata snapshot of 2026-09-12"},"code_links":[{"paper_slug":"statistical-testing-under-distributional","repo_url":"https://github.com/nikolajthams/testing-under-shifts","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2105.10821","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2105.10821"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/nikolajthams/testing-under-shifts","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"unverified":10},"by_repo_kind":{"official":{"samples":10,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"ed98c2fe5184d4dd","entry":"coin","repo":"nikolajthams/testing-under-shifts","repo_kind":"official","path":"experiments/section-5-3/cond-independence-test.py","file_url":"https://github.com/nikolajthams/testing-under-shifts/blob/HEAD/experiments/section-5-3/cond-independence-test.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ed98c2fe5184d4dd"}},{"code_sha256_prefix":"cf2b5109398e3ea9","entry":"e","repo":"nikolajthams/testing-under-shifts","repo_kind":"official","path":"experiments/section-5-1/choice-m-m-tuning.py","file_url":"https://github.com/nikolajthams/testing-under-shifts/blob/HEAD/experiments/section-5-1/choice-m-m-tuning.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"cf2b5109398e3ea9"}},{"code_sha256_prefix":"c23060621f3cbf10","entry":"gen_data","repo":"nikolajthams/testing-under-shifts","repo_kind":"official","path":"experiments/section-5-4/experiment-dormant-binary.py","file_url":"https://github.com/nikolajthams/testing-under-shifts/blob/HEAD/experiments/section-5-4/experiment-dormant-binary.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c23060621f3cbf10"}},{"code_sha256_prefix":"ad8b76a0243a7a73","entry":"gen_params","repo":"nikolajthams/testing-under-shifts","repo_kind":"official","path":"experiments/section-5-4/experiment-dormant-binary.py","file_url":"https://github.com/nikolajthams/testing-under-shifts/blob/HEAD/experiments/section-5-4/experiment-dormant-binary.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ad8b76a0243a7a73"}},{"code_sha256_prefix":"8da96ea2b5bb8a12","entry":"get_lik","repo":"nikolajthams/testing-under-shifts","repo_kind":"official","path":"experiments/section-5-6/experiment-icp.py","file_url":"https://github.com/nikolajthams/testing-under-shifts/blob/HEAD/experiments/section-5-6/experiment-icp.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"8da96ea2b5bb8a12"}},{"code_sha256_prefix":"8b88f06b682fb454","entry":"get_weight","repo":"nikolajthams/testing-under-shifts","repo_kind":"official","path":"experiments/section-5-5/compare-to-ipw.py","file_url":"https://github.com/nikolajthams/testing-under-shifts/blob/HEAD/experiments/section-5-5/compare-to-ipw.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"8b88f06b682fb454"}},{"code_sha256_prefix":"3f0261e0dd676ac0","entry":"scm","repo":"nikolajthams/testing-under-shifts","repo_kind":"official","path":"experiments/section-5-1/choice-m-m-tuning.py","file_url":"https://github.com/nikolajthams/testing-under-shifts/blob/HEAD/experiments/section-5-1/choice-m-m-tuning.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"3f0261e0dd676ac0"}},{"code_sha256_prefix":"ca5ee286e7886de0","entry":"u","repo":"nikolajthams/testing-under-shifts","repo_kind":"official","path":"experiments/section-5-1/choice-sigma.py","file_url":"https://github.com/nikolajthams/testing-under-shifts/blob/HEAD/experiments/section-5-1/choice-sigma.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ca5ee286e7886de0"}},{"code_sha256_prefix":"a6813cdbc985f7c8","entry":"u","repo":"nikolajthams/testing-under-shifts","repo_kind":"official","path":"experiments/section-5-4/experiment-dormant-nonparametric.py","file_url":"https://github.com/nikolajthams/testing-under-shifts/blob/HEAD/experiments/section-5-4/experiment-dormant-nonparametric.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a6813cdbc985f7c8"}},{"code_sha256_prefix":"a5475f49612ef323","entry":"weight","repo":"nikolajthams/testing-under-shifts","repo_kind":"official","path":"experiments/section-5-1/choice-m-m-tuning.py","file_url":"https://github.com/nikolajthams/testing-under-shifts/blob/HEAD/experiments/section-5-1/choice-m-m-tuning.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a5475f49612ef323"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}