{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/contiformer-continuous-time-transformer-for-1","title":"ContiFormer: Continuous-Time Transformer for Irregular Time Series Modeling","arxiv_id":"2402.10635","date":"2024-02-16","proceeding":"NeurIPS 2023 11","authors":["Yuqi Chen","Kan Ren","Yansen Wang","Yuchen Fang","Weiwei Sun","Dongsheng Li"],"abstract":"Modeling continuous-time dynamics on irregular time series is critical to account for data evolution and correlations that occur continuously. Traditional methods including recurrent neural networks or Transformer models leverage inductive bias via powerful neural architectures to capture complex patterns. However, due to their discrete characteristic, they have limitations in generalizing to continuous-time data paradigms. Though neural ordinary differential equations (Neural ODEs) and their variants have shown promising results in dealing with irregular time series, they often fail to capture the intricate correlations within these sequences. It is challenging yet demanding to concurrently model the relationship between input data points and capture the dynamic changes of the continuous-time system. To tackle this problem, we propose ContiFormer that extends the relation modeling of vanilla Transformer to the continuous-time domain, which explicitly incorporates the modeling abilities of continuous dynamics of Neural ODEs with the attention mechanism of Transformers. We mathematically characterize the expressive power of ContiFormer and illustrate that, by curated designs of function hypothesis, many Transformer variants specialized in irregular time series modeling can be covered as a special case of ContiFormer. A wide range of experiments on both synthetic and real-world datasets have illustrated the superior modeling capacities and prediction performance of ContiFormer on irregular time series data. The project link is https://seqml.github.io/contiformer/.","url_abs":"https://arxiv.org/abs/2402.10635v1","url_pdf":"https://arxiv.org/pdf/2402.10635v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"contiformer-continuous-time-transformer-for-1","repo_url":"https://github.com/microsoft/SeqML/tree/main/ContiFormer","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"inductive-bias","task_name":"Inductive Bias"},{"task_slug":"irregular-time-series","task_name":"Irregular Time Series"},{"task_slug":"time-series-1","task_name":"Time Series"}],"methods":[{"method_slug":"absolute-position-encodings","method_name":"Absolute Position Encodings"},{"method_slug":"adam","method_name":"Adam"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"bpe","method_name":"BPE"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"label-smoothing","method_name":"Label Smoothing"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"position-wise-feed-forward-layer","method_name":"Position-Wise Feed-Forward Layer"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"transformer","method_name":"Transformer"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2402.10635","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2402.10635"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/microsoft/SeqML/tree/main/ContiFormer","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":8,"unverified":4},"by_repo_kind":{"official":{"samples":12,"ran":8,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"9587f808deb9dbc3","entry":"accuracy","repo":"microsoft/SeqML","repo_kind":"official","path":"IRENE/utils.py","file_url":"https://github.com/microsoft/SeqML/blob/HEAD/IRENE/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"9587f808deb9dbc3"}},{"code_sha256_prefix":"457c048bcec55e19","entry":"accuracy","repo":"microsoft/SeqML","repo_kind":"official","path":"SIMPLE/utils.py","file_url":"https://github.com/microsoft/SeqML/blob/HEAD/SIMPLE/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"457c048bcec55e19"}},{"code_sha256_prefix":"5701e7af59064c38","entry":"check_file_number","repo":"microsoft/SeqML","repo_kind":"official","path":"SIMPLE/utils.py","file_url":"https://github.com/microsoft/SeqML/blob/HEAD/SIMPLE/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"5701e7af59064c38"}},{"code_sha256_prefix":"2d05cbfe6203f265","entry":"dataset_with_indices","repo":"microsoft/SeqML","repo_kind":"official","path":"IRENE/dataset.py","file_url":"https://github.com/microsoft/SeqML/blob/HEAD/IRENE/dataset.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"2d05cbfe6203f265"}},{"code_sha256_prefix":"dc2478e15ade0979","entry":"get_logger","repo":"microsoft/SeqML","repo_kind":"official","path":"ContiFormer/spiral.py","file_url":"https://github.com/microsoft/SeqML/blob/HEAD/ContiFormer/spiral.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"dc2478e15ade0979"}},{"code_sha256_prefix":"c71b4e0700598afa","entry":"get_training_dataloader","repo":"microsoft/SeqML","repo_kind":"official","path":"IRENE/utils.py","file_url":"https://github.com/microsoft/SeqML/blob/HEAD/IRENE/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c71b4e0700598afa"}},{"code_sha256_prefix":"622beaf8dfaa941d","entry":"log_normal_pdf","repo":"microsoft/SeqML","repo_kind":"official","path":"ContiFormer/spiral.py","file_url":"https://github.com/microsoft/SeqML/blob/HEAD/ContiFormer/spiral.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"622beaf8dfaa941d"}},{"code_sha256_prefix":"5ba9d747c5e26e18","entry":"walkFile","repo":"microsoft/SeqML","repo_kind":"official","path":"SIMPLE/utils.py","file_url":"https://github.com/microsoft/SeqML/blob/HEAD/SIMPLE/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"5ba9d747c5e26e18"}},{"code_sha256_prefix":"0a798dd543d533e1","entry":"dataset_with_indices","repo":"microsoft/SeqML","repo_kind":"official","path":"SIMPLE/data_domainbed.py","file_url":"https://github.com/microsoft/SeqML/blob/HEAD/SIMPLE/data_domainbed.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"0a798dd543d533e1"}},{"code_sha256_prefix":"94ca80370ce2e5b9","entry":"generate_spiral2d","repo":"microsoft/SeqML","repo_kind":"official","path":"ContiFormer/spiral.py","file_url":"https://github.com/microsoft/SeqML/blob/HEAD/ContiFormer/spiral.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"94ca80370ce2e5b9"}},{"code_sha256_prefix":"965d0507e2b1e068","entry":"get_network","repo":"microsoft/SeqML","repo_kind":"official","path":"IRENE/utils.py","file_url":"https://github.com/microsoft/SeqML/blob/HEAD/IRENE/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"965d0507e2b1e068"}},{"code_sha256_prefix":"1213aec634167d4b","entry":"look_up","repo":"microsoft/SeqML","repo_kind":"official","path":"IRENE/ensemble.py","file_url":"https://github.com/microsoft/SeqML/blob/HEAD/IRENE/ensemble.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"1213aec634167d4b"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}