{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/ensemble-of-averages-improving-model","title":"Ensemble of Averages: Improving Model Selection and Boosting Performance in Domain Generalization","arxiv_id":"2110.10832","date":"2021-10-21","proceeding":null,"authors":["Devansh Arpit","Huan Wang","Yingbo Zhou","Caiming Xiong"],"abstract":"In Domain Generalization (DG) settings, models trained independently on a given set of training domains have notoriously chaotic performance on distribution shifted test domains, and stochasticity in optimization (e.g. seed) plays a big role. This makes deep learning models unreliable in real world settings. We first show that this chaotic behavior exists even along the training optimization trajectory of a single model, and propose a simple model averaging protocol that both significantly boosts domain generalization and diminishes the impact of stochasticity by improving the rank correlation between the in-domain validation accuracy and out-domain test accuracy, which is crucial for reliable early stopping. Taking advantage of our observation, we show that instead of ensembling unaveraged models (that is typical in practice), ensembling moving average models (EoA) from independent runs further boosts performance. We theoretically explain the boost in performance of ensembling and model averaging by adapting the well known Bias-Variance trade-off to the domain generalization setting. On the DomainBed benchmark, when using a pre-trained ResNet-50, this ensemble of averages achieves an average of $68.0\\%$, beating vanilla ERM (w/o averaging/ensembling) by $\\sim 4\\%$, and when using a pre-trained RegNetY-16GF, achieves an average of $76.6\\%$, beating vanilla ERM by $6\\%$. Our code is available at \\url{https://github.com/salesforce/ensemble-of-averages}.","url_abs":"https://arxiv.org/abs/2110.10832v4","url_pdf":"https://arxiv.org/pdf/2110.10832v4.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"ensemble-of-averages-improving-model","repo_url":"https://github.com/salesforce/ensemble-of-averages","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"domain-generalization","task_name":"Domain Generalization"},{"task_slug":"model-selection","task_name":"Model Selection"}],"methods":[{"method_slug":"1x1-convolution","method_name":"1x1 Convolution"},{"method_slug":"average-pooling","method_name":"Average Pooling"},{"method_slug":"batch-normalization","method_name":"Batch Normalization"},{"method_slug":"bottleneck-residual-block","method_name":"Bottleneck Residual Block"},{"method_slug":"convolution","method_name":"Convolution"},{"method_slug":"global-average-pooling","method_name":"Global Average Pooling"},{"method_slug":"kaiming-initialization","method_name":"Kaiming Initialization"},{"method_slug":"max-pooling","method_name":"Max Pooling"},{"method_slug":"relu","method_name":"ReLU"},{"method_slug":"residual-block","method_name":"Residual Block"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"test","method_name":"Test"}],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/domain-generalization-on-domainnet","task":"Domain Generalization","dataset":"DomainNet","model":"Ensemble of Averages (RegNetY-16GF)","rank_in_archive_order":10,"of":38,"metrics":{"Average Accuracy":"60.9"},"uses_additional_data":true},{"leaderboard":"/sota/domain-generalization-on-domainnet","task":"Domain Generalization","dataset":"DomainNet","model":"Ensemble of Averages (ResNeXt-50 32x4d)","rank_in_archive_order":17,"of":38,"metrics":{"Average Accuracy":"54.6"},"uses_additional_data":true},{"leaderboard":"/sota/domain-generalization-on-domainnet","task":"Domain Generalization","dataset":"DomainNet","model":"Ensemble of Averages (ResNet-50)","rank_in_archive_order":25,"of":38,"metrics":{"Average Accuracy":"47.4"},"uses_additional_data":false},{"leaderboard":"/sota/domain-generalization-on-office-home","task":"Domain Generalization","dataset":"Office-Home","model":"Ensemble of Averages (RegNetY-16GF)","rank_in_archive_order":9,"of":45,"metrics":{"Average Accuracy":"83.9"},"uses_additional_data":false},{"leaderboard":"/sota/domain-generalization-on-office-home","task":"Domain Generalization","dataset":"Office-Home","model":"Ensemble of Averages (ResNeXt-50 32x4d)","rank_in_archive_order":16,"of":45,"metrics":{"Average Accuracy":"80.2"},"uses_additional_data":false},{"leaderboard":"/sota/domain-generalization-on-office-home","task":"Domain Generalization","dataset":"Office-Home","model":"Ensemble of Averages (ResNet-50)","rank_in_archive_order":23,"of":45,"metrics":{"Average Accuracy":"72.5"},"uses_additional_data":false},{"leaderboard":"/sota/domain-generalization-on-pacs-2","task":"Domain Generalization","dataset":"PACS","model":"Ensemble of Averages (RegNetY-16GF)","rank_in_archive_order":13,"of":133,"metrics":{"Average Accuracy":"95.8"},"uses_additional_data":false},{"leaderboard":"/sota/domain-generalization-on-pacs-2","task":"Domain Generalization","dataset":"PACS","model":"Ensemble of Averages (ResNeXt-50 32x4d)","rank_in_archive_order":16,"of":133,"metrics":{"Average Accuracy":"93.2"},"uses_additional_data":false},{"leaderboard":"/sota/domain-generalization-on-pacs-2","task":"Domain Generalization","dataset":"PACS","model":"Ensemble of Averages (ResNet-50)","rank_in_archive_order":25,"of":133,"metrics":{"Average Accuracy":"88.6"},"uses_additional_data":false},{"leaderboard":"/sota/domain-generalization-on-terraincognita","task":"Domain Generalization","dataset":"TerraIncognita","model":"Ensemble of Averages (RegNetY-16GF)","rank_in_archive_order":5,"of":30,"metrics":{"Average Accuracy":"61.1"},"uses_additional_data":false},{"leaderboard":"/sota/domain-generalization-on-terraincognita","task":"Domain Generalization","dataset":"TerraIncognita","model":"Ensemble of Averages (ResNeXt-50 32x4d)","rank_in_archive_order":13,"of":30,"metrics":{"Average Accuracy":"55.2"},"uses_additional_data":false},{"leaderboard":"/sota/domain-generalization-on-terraincognita","task":"Domain Generalization","dataset":"TerraIncognita","model":"Ensemble of Averages (ResNet-50)","rank_in_archive_order":18,"of":30,"metrics":{"Average Accuracy":"52.3"},"uses_additional_data":false},{"leaderboard":"/sota/domain-generalization-on-vlcs","task":"Domain Generalization","dataset":"VLCS","model":"Ensemble of Averages (RegNetY-16GF)","rank_in_archive_order":17,"of":37,"metrics":{"Average Accuracy":"81.1"},"uses_additional_data":false},{"leaderboard":"/sota/domain-generalization-on-vlcs","task":"Domain Generalization","dataset":"VLCS","model":"Ensemble of Averages (ResNeXt-50 32x4d)","rank_in_archive_order":18,"of":37,"metrics":{"Average Accuracy":"80.4"},"uses_additional_data":false},{"leaderboard":"/sota/domain-generalization-on-vlcs","task":"Domain Generalization","dataset":"VLCS","model":"Ensemble of Averages (ResNet-50)","rank_in_archive_order":30,"of":37,"metrics":{"Average Accuracy":"79.1"},"uses_additional_data":false}],"syntology":{"syntology_url":null,"atlas_url":"https://app.syntology.ai/?focus=2110.10832","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2110.10832"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/salesforce/ensemble-of-averages","reach":null}],"summary":{"ran_fixture":1,"ran_draft_wrong":1},"by_repo_kind":{"official":{"samples":2,"ran":2,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":2,"samples":[{"code_sha256_prefix":"5e5f34bd72149c91","entry":"accuracy","repo":"salesforce/ensemble-of-averages","repo_kind":"official","path":"domainbed/EoA.py","file_url":"https://github.com/salesforce/ensemble-of-averages/blob/HEAD/domainbed/EoA.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"5e5f34bd72149c91"}},{"code_sha256_prefix":"b617416b75fcbe19","entry":"rename_dict","repo":"salesforce/ensemble-of-averages","repo_kind":"official","path":"domainbed/EoA.py","file_url":"https://github.com/salesforce/ensemble-of-averages/blob/HEAD/domainbed/EoA.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"b617416b75fcbe19"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}