{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/freetickets-accurate-robust-and-efficient","title":"Deep Ensembling with No Overhead for either Training or Testing: The All-Round Blessings of Dynamic Sparsity","arxiv_id":"2106.14568","date":"2021-06-28","proceeding":"ICLR 2022 4","authors":["Shiwei Liu","Tianlong Chen","Zahra Atashgahi","Xiaohan Chen","Ghada Sokar","Elena Mocanu","Mykola Pechenizkiy","Zhangyang Wang","Decebal Constantin Mocanu"],"abstract":"The success of deep ensembles on improving predictive performance, uncertainty estimation, and out-of-distribution robustness has been extensively studied in the machine learning literature. Albeit the promising results, naively training multiple deep neural networks and combining their predictions at inference leads to prohibitive computational costs and memory requirements. Recently proposed efficient ensemble approaches reach the performance of the traditional deep ensembles with significantly lower costs. However, the training resources required by these approaches are still at least the same as training a single dense model. In this work, we draw a unique connection between sparse neural network training and deep ensembles, yielding a novel efficient ensemble learning framework called FreeTickets. Instead of training multiple dense networks and averaging them, we directly train sparse subnetworks from scratch and extract diverse yet accurate subnetworks during this efficient, sparse-to-sparse training. Our framework, FreeTickets, is defined as the ensemble of these relatively cheap sparse subnetworks. Despite being an ensemble method, FreeTickets has even fewer parameters and training FLOPs than a single dense model. This seemingly counter-intuitive outcome is due to the ultra training/inference efficiency of dynamic sparse training. FreeTickets surpasses the dense baseline in all the following criteria: prediction accuracy, uncertainty estimation, out-of-distribution (OoD) robustness, as well as efficiency for both training and inference. Impressively, FreeTickets outperforms the naive deep ensemble with ResNet50 on ImageNet using around only 1/5 of the training FLOPs required by the latter. We have released our source code at https://github.com/VITA-Group/FreeTickets.","url_abs":"https://arxiv.org/abs/2106.14568v4","url_pdf":"https://arxiv.org/pdf/2106.14568v4.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"freetickets-accurate-robust-and-efficient","repo_url":"https://github.com/Shiweiliuiiiiiii/FreeTickets","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok"}},{"paper_slug":"freetickets-accurate-robust-and-efficient","repo_url":"https://github.com/vita-group/freetickets","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"all","task_name":"All"},{"task_slug":"ensemble-learning","task_name":"Ensemble Learning"},{"task_slug":"image-classification","task_name":"Image Classification"}],"methods":[{"method_slug":"deep-ensembles","method_name":"Deep Ensembles"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2106.14568","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2106.14568"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/VITA-Group/FreeTickets","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/Shiweiliuiiiiiii/FreeTickets","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/vita-group/freetickets","reach":null}],"summary":{"ran_draft_wrong":2,"ran":1,"ran_fixture":1,"unverified":1},"by_repo_kind":{"official":{"samples":5,"ran":4,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":5,"samples":[{"code_sha256_prefix":"2cfdf5516f63cbbf","entry":"sorted_nicely","repo":"VITA-Group/FreeTickets","repo_kind":"official","path":"ensemble_freetickets.py","file_url":"https://github.com/VITA-Group/FreeTickets/blob/HEAD/ensemble_freetickets.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":2,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"2cfdf5516f63cbbf"}},{"code_sha256_prefix":"4bb21a77b180d6b4","entry":"Masking","repo":"vita-group/freetickets","repo_kind":"official","path":"sparselearning/core.py","file_url":"https://github.com/vita-group/freetickets/blob/HEAD/sparselearning/core.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"4bb21a77b180d6b4"}},{"code_sha256_prefix":"64f8ee64a2582932","entry":"evaluate_tsne","repo":"VITA-Group/FreeTickets","repo_kind":"official","path":"ensemble_freetickets.py","file_url":"https://github.com/VITA-Group/FreeTickets/blob/HEAD/ensemble_freetickets.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"64f8ee64a2582932"}},{"code_sha256_prefix":"7b0198284644a14e","entry":"extract_prediction","repo":"VITA-Group/FreeTickets","repo_kind":"official","path":"ensemble_freetickets.py","file_url":"https://github.com/VITA-Group/FreeTickets/blob/HEAD/ensemble_freetickets.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"7b0198284644a14e"}},{"code_sha256_prefix":"b4b09ad38c701730","entry":"evaluate","repo":"VITA-Group/FreeTickets","repo_kind":"official","path":"main_DST.py","file_url":"https://github.com/VITA-Group/FreeTickets/blob/HEAD/main_DST.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"b4b09ad38c701730"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}