{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/assaying-out-of-distribution-generalization","title":"Assaying Out-Of-Distribution Generalization in Transfer Learning","arxiv_id":"2207.09239","date":"2022-07-19","proceeding":null,"authors":["Florian Wenzel","Andrea Dittadi","Peter Vincent Gehler","Carl-Johann Simon-Gabriel","Max Horn","Dominik Zietlow","David Kernert","Chris Russell","Thomas Brox","Bernt Schiele","Bernhard Schölkopf","Francesco Locatello"],"abstract":"Since out-of-distribution generalization is a generally ill-posed problem, various proxy targets (e.g., calibration, adversarial robustness, algorithmic corruptions, invariance across shifts) were studied across different research programs resulting in different recommendations. While sharing the same aspirational goal, these approaches have never been tested under the same experimental conditions on real data. In this paper, we take a unified view of previous work, highlighting message discrepancies that we address empirically, and providing recommendations on how to measure the robustness of a model and how to improve it. To this end, we collect 172 publicly available dataset pairs for training and out-of-distribution evaluation of accuracy, calibration error, adversarial attacks, environment invariance, and synthetic corruptions. We fine-tune over 31k networks, from nine different architectures in the many- and few-shot setting. Our findings confirm that in- and out-of-distribution accuracies tend to increase jointly, but show that their relation is largely dataset-dependent, and in general more nuanced and more complex than posited by previous, smaller scale studies.","url_abs":"https://arxiv.org/abs/2207.09239v2","url_pdf":"https://arxiv.org/pdf/2207.09239v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"assaying-out-of-distribution-generalization","repo_url":"https://github.com/amazon-research/assaying-ood","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"adversarial-robustness","task_name":"Adversarial Robustness"},{"task_slug":"out-of-distribution-generalization","task_name":"Out-of-Distribution Generalization"},{"task_slug":"transfer-learning","task_name":"Transfer Learning"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2207.09239","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2207.09239"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/amazon-research/assaying-ood","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"unverified":5},"by_repo_kind":{"official":{"samples":5,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"715cd49ccf6053bc","entry":"adapt_batch_size","repo":"amazon-research/assaying-ood","repo_kind":"official","path":"src/ood_inspector/evaluations.py","file_url":"https://github.com/amazon-research/assaying-ood/blob/HEAD/src/ood_inspector/evaluations.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"715cd49ccf6053bc"}},{"code_sha256_prefix":"c6ae838a3198d991","entry":"flatten_dict_to_scoped_json","repo":"amazon-research/assaying-ood","repo_kind":"official","path":"src/ood_inspector/utils.py","file_url":"https://github.com/amazon-research/assaying-ood/blob/HEAD/src/ood_inspector/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"c6ae838a3198d991"}},{"code_sha256_prefix":"afa07d48a94312c4","entry":"get_device_from_module","repo":"amazon-research/assaying-ood","repo_kind":"official","path":"src/ood_inspector/utils.py","file_url":"https://github.com/amazon-research/assaying-ood/blob/HEAD/src/ood_inspector/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"afa07d48a94312c4"}},{"code_sha256_prefix":"8f30254f2c9ef39b","entry":"load_base_transform","repo":"amazon-research/assaying-ood","repo_kind":"official","path":"src/ood_inspector/transform.py","file_url":"https://github.com/amazon-research/assaying-ood/blob/HEAD/src/ood_inspector/transform.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"8f30254f2c9ef39b"}},{"code_sha256_prefix":"2e709200c6f9cd80","entry":"track_input_of_layer","repo":"amazon-research/assaying-ood","repo_kind":"official","path":"src/ood_inspector/utils.py","file_url":"https://github.com/amazon-research/assaying-ood/blob/HEAD/src/ood_inspector/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"2e709200c6f9cd80"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}