{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/wild-time-a-benchmark-of-in-the-wild","title":"Wild-Time: A Benchmark of in-the-Wild Distribution Shift over Time","arxiv_id":"2211.14238","date":"2022-11-25","proceeding":null,"authors":["Huaxiu Yao","Caroline Choi","Bochuan Cao","Yoonho Lee","Pang Wei Koh","Chelsea Finn"],"abstract":"Distribution shift occurs when the test distribution differs from the training distribution, and it can considerably degrade performance of machine learning models deployed in the real world. Temporal shifts -- distribution shifts arising from the passage of time -- often occur gradually and have the additional structure of timestamp metadata. By leveraging timestamp metadata, models can potentially learn from trends in past distribution shifts and extrapolate into the future. While recent works have studied distribution shifts, temporal shifts remain underexplored. To address this gap, we curate Wild-Time, a benchmark of 5 datasets that reflect temporal distribution shifts arising in a variety of real-world applications, including patient prognosis and news classification. On these datasets, we systematically benchmark 13 prior approaches, including methods in domain generalization, continual learning, self-supervised learning, and ensemble learning. We use two evaluation strategies: evaluation with a fixed time split (Eval-Fix) and evaluation with a data stream (Eval-Stream). Eval-Fix, our primary evaluation strategy, aims to provide a simple evaluation protocol, while Eval-Stream is more realistic for certain real-world applications. Under both evaluation strategies, we observe an average performance drop of 20% from in-distribution to out-of-distribution data. Existing methods are unable to close this gap. Code is available at https://wild-time.github.io/.","url_abs":"https://arxiv.org/abs/2211.14238v2","url_pdf":"https://arxiv.org/pdf/2211.14238v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"wild-time-a-benchmark-of-in-the-wild","repo_url":"https://github.com/huaxiuyao/wild-time","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"continual-learning","task_name":"Continual Learning"},{"task_slug":"domain-generalization","task_name":"Domain Generalization"},{"task_slug":"ensemble-learning","task_name":"Ensemble Learning"},{"task_slug":"news-classification","task_name":"News Classification"},{"task_slug":"prognosis","task_name":"Prognosis"},{"task_slug":"self-supervised-learning","task_name":"Self-Supervised Learning"}],"methods":[{"method_slug":"test","method_name":"Test"}],"datasets_introduced":[{"slug":"wild-time","name":"Wild-Time","full_name":""}],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2211.14238","atlas_url":"https://app.syntology.ai/?focus=2211.14238","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2211.14238"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/fMoW/dataset","reach":{"status":"ok","spdx":"NOASSERTION"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/huaxiuyao/wild-time","reach":null}],"summary":{"unverified":2},"by_repo_kind":{"official":{"samples":2,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"beca3cd43469c27a","entry":"init","repo":"huaxiuyao/wild-time","repo_kind":"official","path":"wildtime/baseline_trainer.py","file_url":"https://github.com/huaxiuyao/wild-time/blob/HEAD/wildtime/baseline_trainer.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"beca3cd43469c27a"}},{"code_sha256_prefix":"60f3099a9cae2b5e","entry":"trainer_init","repo":"huaxiuyao/wild-time","repo_kind":"official","path":"wildtime/baseline_trainer.py","file_url":"https://github.com/huaxiuyao/wild-time/blob/HEAD/wildtime/baseline_trainer.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"60f3099a9cae2b5e"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}