{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/invariance-principle-meets-information","title":"Invariance Principle Meets Information Bottleneck for Out-of-Distribution Generalization","arxiv_id":"2106.06607","date":"2021-06-11","proceeding":"NeurIPS 2021 12","authors":["Kartik Ahuja","Ethan Caballero","Dinghuai Zhang","Jean-Christophe Gagnon-Audet","Yoshua Bengio","Ioannis Mitliagkas","Irina Rish"],"abstract":"The invariance principle from causality is at the heart of notable approaches such as invariant risk minimization (IRM) that seek to address out-of-distribution (OOD) generalization failures. Despite the promising theory, invariance principle-based approaches fail in common classification tasks, where invariant (causal) features capture all the information about the label. Are these failures due to the methods failing to capture the invariance? Or is the invariance principle itself insufficient? To answer these questions, we revisit the fundamental assumptions in linear regression tasks, where invariance-based approaches were shown to provably generalize OOD. In contrast to the linear regression tasks, we show that for linear classification tasks we need much stronger restrictions on the distribution shifts, or otherwise OOD generalization is impossible. Furthermore, even with appropriate restrictions on distribution shifts in place, we show that the invariance principle alone is insufficient. We prove that a form of the information bottleneck constraint along with invariance helps address key failures when invariant features capture all the information about the label and also retains the existing success when they do not. We propose an approach that incorporates both of these principles and demonstrate its effectiveness in several experiments.","url_abs":"https://arxiv.org/abs/2106.06607v2","url_pdf":"https://arxiv.org/pdf/2106.06607v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"invariance-principle-meets-information","repo_url":"https://github.com/facebookresearch/DomainBed","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"invariance-principle-meets-information","repo_url":"https://github.com/ahujak/IB-IRM","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"out-of-distribution-generalization","task_name":"Out-of-Distribution Generalization"},{"task_slug":"regression-1","task_name":"regression"}],"methods":[{"method_slug":"linear-regression","method_name":"Linear Regression"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2106.06607","atlas_url":"https://app.syntology.ai/?focus=2106.06607","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2106.06607"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/facebookresearch/DomainBed","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/ahujak/IB-IRM","reach":null}],"summary":{"ran_fixture":1,"ran_draft_wrong":2,"unverified":3},"by_repo_kind":{"official":{"samples":3,"ran":3,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":6,"samples":[{"code_sha256_prefix":"70bca7290fc6b3be","entry":"make_environment","repo":"ahujak/IB-IRM","repo_kind":"official","path":"AC-CMNIST/experiments/cmnist/irm_baseline/val_main.py","file_url":"https://github.com/ahujak/IB-IRM/blob/HEAD/AC-CMNIST/experiments/cmnist/irm_baseline/val_main.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"70bca7290fc6b3be"}},{"code_sha256_prefix":"cbb762a6632ddabc","entry":"mean_accuracy","repo":"ahujak/IB-IRM","repo_kind":"official","path":"AC-CMNIST/experiments/cmnist/irm_baseline/val_main.py","file_url":"https://github.com/ahujak/IB-IRM/blob/HEAD/AC-CMNIST/experiments/cmnist/irm_baseline/val_main.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"cbb762a6632ddabc"}},{"code_sha256_prefix":"b94e0dbe411acd7f","entry":"mean_nll","repo":"ahujak/IB-IRM","repo_kind":"official","path":"AC-CMNIST/experiments/cmnist/irm_baseline/val_main.py","file_url":"https://github.com/ahujak/IB-IRM/blob/HEAD/AC-CMNIST/experiments/cmnist/irm_baseline/val_main.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"b94e0dbe411acd7f"}},{"code_sha256_prefix":"b0bc80b1655a6802","entry":"get_algorithm_class","repo":null,"repo_kind":null,"path":null,"file_url":null,"link_basis":"identical_code_first_harvested_elsewhere","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":null,"inline_ok":false,"mcp_get_code":{"code_sha256":"b0bc80b1655a6802"}},{"code_sha256_prefix":"d0ea85d74c20dea9","entry":"get_dataset_class","repo":null,"repo_kind":null,"path":null,"file_url":null,"link_basis":"identical_code_first_harvested_elsewhere","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":null,"inline_ok":false,"mcp_get_code":{"code_sha256":"d0ea85d74c20dea9"}},{"code_sha256_prefix":"73f32252eedba6a4","entry":"num_environments","repo":null,"repo_kind":null,"path":null,"file_url":null,"link_basis":"identical_code_first_harvested_elsewhere","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":null,"inline_ok":false,"mcp_get_code":{"code_sha256":"73f32252eedba6a4"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}