{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/cleva-compass-a-continual-learning-evaluation","title":"CLEVA-Compass: A Continual Learning EValuation Assessment Compass to Promote Research Transparency and Comparability","arxiv_id":"2110.03331","date":"2021-10-07","proceeding":"ICLR 2022 4","authors":["Martin Mundt","Steven Lang","Quentin Delfosse","Kristian Kersting"],"abstract":"What is the state of the art in continual machine learning? Although a natural question for predominant static benchmarks, the notion to train systems in a lifelong manner entails a plethora of additional challenges with respect to set-up and evaluation. The latter have recently sparked a growing amount of critiques on prominent algorithm-centric perspectives and evaluation protocols being too narrow, resulting in several attempts at constructing guidelines in favor of specific desiderata or arguing against the validity of prevalent assumptions. In this work, we depart from this mindset and argue that the goal of a precise formulation of desiderata is an ill-posed one, as diverse applications may always warrant distinct scenarios. Instead, we introduce the Continual Learning EValuation Assessment Compass: the CLEVA-Compass. The compass provides the visual means to both identify how approaches are practically reported and how works can simultaneously be contextualized in the broader literature landscape. In addition to promoting compact specification in the spirit of recent replication trends, it thus provides an intuitive chart to understand the priorities of individual systems, where they resemble each other, and what elements are missing towards a fair comparison.","url_abs":"https://arxiv.org/abs/2110.03331v2","url_pdf":"https://arxiv.org/pdf/2110.03331v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"cleva-compass-a-continual-learning-evaluation","repo_url":"https://github.com/ml-research/CLEVA-Compass","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"none","reach":null}],"tasks":[{"task_slug":"continual-learning","task_name":"Continual Learning"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2110.03331","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2110.03331"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/ml-research/CLEVA-Compass","reach":null}],"summary":{"ran":3,"ran_draft_wrong":5,"unverified":2},"by_repo_kind":{"official":{"samples":10,"ran":8,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"070b06b058ccd060","entry":"CompassEntry","repo":"ml-research/CLEVA-Compass","repo_kind":"official","path":"create_compass.py","file_url":"https://github.com/ml-research/CLEVA-Compass/blob/HEAD/create_compass.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"070b06b058ccd060"}},{"code_sha256_prefix":"5704f6fbebf03b56","entry":"InnerLevel","repo":"ml-research/CLEVA-Compass","repo_kind":"official","path":"create_compass.py","file_url":"https://github.com/ml-research/CLEVA-Compass/blob/HEAD/create_compass.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"5704f6fbebf03b56"}},{"code_sha256_prefix":"ba68a5d5e9e187b3","entry":"OuterLevel","repo":"ml-research/CLEVA-Compass","repo_kind":"official","path":"create_compass.py","file_url":"https://github.com/ml-research/CLEVA-Compass/blob/HEAD/create_compass.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ba68a5d5e9e187b3"}},{"code_sha256_prefix":"7893ea67beb14fee","entry":"fill_template","repo":"ml-research/CLEVA-Compass","repo_kind":"official","path":"create_compass.py","file_url":"https://github.com/ml-research/CLEVA-Compass/blob/HEAD/create_compass.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"7893ea67beb14fee"}},{"code_sha256_prefix":"3ff69caeb20a8e83","entry":"insert_legend","repo":"ml-research/CLEVA-Compass","repo_kind":"official","path":"create_compass.py","file_url":"https://github.com/ml-research/CLEVA-Compass/blob/HEAD/create_compass.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"3ff69caeb20a8e83"}},{"code_sha256_prefix":"977ff51a91af0715","entry":"insert_number_of_methods","repo":"ml-research/CLEVA-Compass","repo_kind":"official","path":"create_compass.py","file_url":"https://github.com/ml-research/CLEVA-Compass/blob/HEAD/create_compass.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"977ff51a91af0715"}},{"code_sha256_prefix":"302927553501b70e","entry":"mapcolor","repo":"ml-research/CLEVA-Compass","repo_kind":"official","path":"create_compass.py","file_url":"https://github.com/ml-research/CLEVA-Compass/blob/HEAD/create_compass.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"302927553501b70e"}},{"code_sha256_prefix":"982ca0b928b57cd9","entry":"update_background","repo":"ml-research/CLEVA-Compass","repo_kind":"official","path":"cleva_gui.py","file_url":"https://github.com/ml-research/CLEVA-Compass/blob/HEAD/cleva_gui.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"982ca0b928b57cd9"}},{"code_sha256_prefix":"e93ca7234a4de368","entry":"insert_inner_level","repo":"ml-research/CLEVA-Compass","repo_kind":"official","path":"create_compass.py","file_url":"https://github.com/ml-research/CLEVA-Compass/blob/HEAD/create_compass.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e93ca7234a4de368"}},{"code_sha256_prefix":"a756bab4d915ae92","entry":"insert_outer_level","repo":"ml-research/CLEVA-Compass","repo_kind":"official","path":"create_compass.py","file_url":"https://github.com/ml-research/CLEVA-Compass/blob/HEAD/create_compass.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a756bab4d915ae92"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}