{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/improving-online-continual-learning","title":"Improving Online Continual Learning Performance and Stability with Temporal Ensembles","arxiv_id":"2306.16817","date":"2023-06-29","proceeding":null,"authors":["Albin Soutif--Cormerais","Antonio Carta","Joost Van de Weijer"],"abstract":"Neural networks are very effective when trained on large datasets for a large number of iterations. However, when they are trained on non-stationary streams of data and in an online fashion, their performance is reduced (1) by the online setup, which limits the availability of data, (2) due to catastrophic forgetting because of the non-stationary nature of the data. Furthermore, several recent works (Caccia et al., 2022; Lange et al., 2023) arXiv:2205.13452 showed that replay methods used in continual learning suffer from the stability gap, encountered when evaluating the model continually (rather than only on task boundaries). In this article, we study the effect of model ensembling as a way to improve performance and stability in online continual learning. We notice that naively ensembling models coming from a variety of training tasks increases the performance in online continual learning considerably. Starting from this observation, and drawing inspirations from semi-supervised learning ensembling methods, we use a lightweight temporal ensemble that computes the exponential moving average of the weights (EMA) at test time, and show that it can drastically increase the performance and stability when used in combination with several methods from the literature.","url_abs":"https://arxiv.org/abs/2306.16817v2","url_pdf":"https://arxiv.org/pdf/2306.16817v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"improving-online-continual-learning","repo_url":"https://github.com/albinsou/online_ema","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"continual-learning","task_name":"Continual Learning"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2306.16817","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2306.16817"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/albinsou/online_ema","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran_draft_wrong":2,"unverified":4},"by_repo_kind":{"official":{"samples":6,"ran":2,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"fac5364e2f53c6db","entry":"conv3x3","repo":"albinsou/online_ema","repo_kind":"official","path":"toolkit/models.py","file_url":"https://github.com/albinsou/online_ema/blob/HEAD/toolkit/models.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"fac5364e2f53c6db"}},{"code_sha256_prefix":"ec20f22a185cd708","entry":"conv3x3","repo":"albinsou/online_ema","repo_kind":"official","path":"toolkit/resnet18.py","file_url":"https://github.com/albinsou/online_ema/blob/HEAD/toolkit/resnet18.py","link_basis":"plan_row","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ec20f22a185cd708"}},{"code_sha256_prefix":"34cddd3729d36ed8","entry":"resnet18","repo":"albinsou/online_ema","repo_kind":"official","path":"toolkit/resnet18.py","file_url":"https://github.com/albinsou/online_ema/blob/HEAD/toolkit/resnet18.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"34cddd3729d36ed8"}},{"code_sha256_prefix":"7fc38981a7259751","entry":"resnet32","repo":"albinsou/online_ema","repo_kind":"official","path":"toolkit/models.py","file_url":"https://github.com/albinsou/online_ema/blob/HEAD/toolkit/models.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"7fc38981a7259751"}},{"code_sha256_prefix":"bb36ec350e827808","entry":"simple_cnn","repo":"albinsou/online_ema","repo_kind":"official","path":"toolkit/models.py","file_url":"https://github.com/albinsou/online_ema/blob/HEAD/toolkit/models.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"bb36ec350e827808"}},{"code_sha256_prefix":"3d7a3fde7da6f042","entry":"warmup_lr","repo":"albinsou/online_ema","repo_kind":"official","path":"src/lr_schedule.py","file_url":"https://github.com/albinsou/online_ema/blob/HEAD/src/lr_schedule.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"3d7a3fde7da6f042"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}