{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/checkpoint-ensembles-ensemble-methods-from-a","title":"Checkpoint Ensembles: Ensemble Methods from a Single Training Process","arxiv_id":"1710.03282","date":"2017-10-09","proceeding":null,"authors":["Hugh Chen","Scott Lundberg","Su-In Lee"],"abstract":"We present the checkpoint ensembles method that can learn ensemble models on\na single training process. Although checkpoint ensembles can be applied to any\nparametric iterative learning technique, here we focus on neural networks.\nNeural networks' composable and simple neurons make it possible to capture many\nindividual and interaction effects among features. However, small sample sizes\nand sampling noise may result in patterns in the training data that are not\nrepresentative of the true relationship between the features and the outcome.\nAs a solution, regularization during training is often used (e.g. dropout).\nHowever, regularization is no panacea -- it does not perfectly address\noverfitting. Even with methods like dropout, two methodologies are commonly\nused in practice. First is to utilize a validation set independent to the\ntraining set as a way to decide when to stop training. Second is to use\nensemble methods to further reduce overfitting and take advantage of local\noptima (i.e. averaging over the predictions of several models). In this paper,\nwe explore checkpoint ensembles -- a simple technique that combines these two\nideas in one training process. Checkpoint ensembles improve performance by\naveraging the predictions from \"checkpoints\" of the best models within single\ntraining process. We use three real-world data sets -- text, image, and\nelectronic health record data -- using three prediction models: a vanilla\nneural network, a convolutional neural network, and a long short term memory\nnetwork to show that checkpoint ensembles outperform existing methods: a method\nthat selects a model by minimum validation score, and two methods that average\nmodels by weights. Our results also show that checkpoint ensembles capture a\nportion of the performance gains that traditional ensembles provide.","url_abs":"http://arxiv.org/abs/1710.03282v1","url_pdf":"http://arxiv.org/pdf/1710.03282v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"checkpoint-ensembles-ensemble-methods-from-a","repo_url":"https://github.com/ENSTA-U2IS-AI/torch-uncertainty","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=1710.03282","mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}