{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/is-importance-weighting-incompatible-with-1","title":"Is Importance Weighting Incompatible with Interpolating Classifiers?","arxiv_id":"2112.12986","date":"2021-12-24","proceeding":"ICLR 2022 4","authors":["Ke Alexander Wang","Niladri S. Chatterji","Saminul Haque","Tatsunori Hashimoto"],"abstract":"Importance weighting is a classic technique to handle distribution shifts. However, prior work has presented strong empirical and theoretical evidence demonstrating that importance weights can have little to no effect on overparameterized neural networks. Is importance weighting truly incompatible with the training of overparameterized neural networks? Our paper answers this in the negative. We show that importance weighting fails not because of the overparameterization, but instead, as a result of using exponentially-tailed losses like the logistic or cross-entropy loss. As a remedy, we show that polynomially-tailed losses restore the effects of importance reweighting in correcting distribution shift in overparameterized models. We characterize the behavior of gradient descent on importance weighted polynomially-tailed losses with overparameterized linear models, and theoretically demonstrate the advantage of using polynomially-tailed losses in a label shift setting. Surprisingly, our theory shows that using weights that are obtained by exponentiating the classical unbiased importance weights can improve performance. Finally, we demonstrate the practical value of our analysis with neural network experiments on a subpopulation shift and a label shift dataset. When reweighted, our loss function can outperform reweighted cross-entropy by as much as 9% in test accuracy. Our loss function also gives test accuracies comparable to, or even exceeding, well-tuned state-of-the-art methods for correcting distribution shifts.","url_abs":"https://arxiv.org/abs/2112.12986v2","url_pdf":"https://arxiv.org/pdf/2112.12986v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"is-importance-weighting-incompatible-with-1","repo_url":"https://github.com/keawang/importance-weighting-interpolating-classifiers","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2112.12986","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2112.12986"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/keawang/importance-weighting-interpolating-classifiers","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"unverified":5},"by_repo_kind":{"official":{"samples":5,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"c946a1ff1f518dd9","entry":"get_transforms_list","repo":"keawang/importance-weighting-interpolating-classifiers","repo_kind":"official","path":"src/pl_datamodules/celeba_datamodule.py","file_url":"https://github.com/keawang/importance-weighting-interpolating-classifiers/blob/HEAD/src/pl_datamodules/celeba_datamodule.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c946a1ff1f518dd9"}},{"code_sha256_prefix":"e800928d4f822a38","entry":"size_to_int","repo":"keawang/importance-weighting-interpolating-classifiers","repo_kind":"official","path":"src/architectures/simple_models.py","file_url":"https://github.com/keawang/importance-weighting-interpolating-classifiers/blob/HEAD/src/architectures/simple_models.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e800928d4f822a38"}},{"code_sha256_prefix":"47d36c87f2335145","entry":"split_dataset","repo":"keawang/importance-weighting-interpolating-classifiers","repo_kind":"official","path":"src/datasets/utils.py","file_url":"https://github.com/keawang/importance-weighting-interpolating-classifiers/blob/HEAD/src/datasets/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"47d36c87f2335145"}},{"code_sha256_prefix":"e6e8883a68fd5457","entry":"subsample_groups","repo":"keawang/importance-weighting-interpolating-classifiers","repo_kind":"official","path":"src/pl_datamodules/celeba_datamodule.py","file_url":"https://github.com/keawang/importance-weighting-interpolating-classifiers/blob/HEAD/src/pl_datamodules/celeba_datamodule.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e6e8883a68fd5457"}},{"code_sha256_prefix":"de524736ca317085","entry":"undersampling_schedule","repo":"keawang/importance-weighting-interpolating-classifiers","repo_kind":"official","path":"src/datasets/utils.py","file_url":"https://github.com/keawang/importance-weighting-interpolating-classifiers/blob/HEAD/src/datasets/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"de524736ca317085"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}