{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/diffusion-auto-regressive-transformer-for","title":"Diffusion Auto-regressive Transformer for Effective Self-supervised Time Series Forecasting","arxiv_id":"2410.05711","date":"2024-10-08","proceeding":null,"authors":["Daoyu Wang","Mingyue Cheng","Zhiding Liu","Qi Liu","Enhong Chen"],"abstract":"Self-supervised learning has become a popular and effective approach for enhancing time series forecasting, enabling models to learn universal representations from unlabeled data. However, effectively capturing both the global sequence dependence and local detail features within time series data remains challenging. To address this, we propose a novel generative self-supervised method called TimeDART, denoting Diffusion Auto-regressive Transformer for Time series forecasting. In TimeDART, we treat time series patches as basic modeling units. Specifically, we employ an self-attention based Transformer encoder to model the dependencies of inter-patches. Additionally, we introduce diffusion and denoising mechanisms to capture the detail locality features of intra-patch. Notably, we design a cross-attention-based denoising decoder that allows for adjustable optimization difficulty in the self-supervised task, facilitating more effective self-supervised pre-training. Furthermore, the entire model is optimized in an auto-regressive manner to obtain transferable representations. Extensive experiments demonstrate that TimeDART achieves state-of-the-art fine-tuning performance compared to the most advanced competitive methods in forecasting tasks. Our code is publicly available at https://github.com/Melmaphother/TimeDART.","url_abs":"https://arxiv.org/abs/2410.05711v2","url_pdf":"https://arxiv.org/pdf/2410.05711v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"diffusion-auto-regressive-transformer-for","repo_url":"https://github.com/melmaphother/timedart","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}},{"paper_slug":"diffusion-auto-regressive-transformer-for","repo_url":"https://github.com/mingyue-cheng/crosstimenet","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}},{"paper_slug":"diffusion-auto-regressive-transformer-for","repo_url":"https://github.com/mingyue-cheng/timemae","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":"decoder","task_name":"Decoder"},{"task_slug":"denoising","task_name":"Denoising"},{"task_slug":"self-supervised-learning","task_name":"Self-Supervised Learning"},{"task_slug":"time-series-1","task_name":"Time Series"},{"task_slug":"time-series-forecasting","task_name":"Time Series Forecasting"}],"methods":[{"method_slug":"absolute-position-encodings","method_name":"Absolute Position Encodings"},{"method_slug":"adam","method_name":"Adam"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"bpe","method_name":"BPE"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"diffusion","method_name":"Diffusion"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"label-smoothing","method_name":"Label Smoothing"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"position-wise-feed-forward-layer","method_name":"Position-Wise Feed-Forward Layer"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"transformer","method_name":"Transformer"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":null,"atlas_url":"https://app.syntology.ai/?focus=2410.05711","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2410.05711"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/Melmaphother/TimeDART","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/mingyue-cheng/crosstimenet","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/mingyue-cheng/timemae","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/melmaphother/timedart","reach":{"status":"ok"}}],"summary":{"ran":20,"ran_violates":2,"ran_honours":1,"ran_draft_wrong":1,"ran_fixture":2,"unverified":5},"by_repo_kind":{"official":{"samples":20,"ran":17,"repositories":1},"listed":{"samples":11,"ran":9,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":31,"samples":[{"code_sha256_prefix":"d8cfebf163c1dda7","entry":"AbsolutePositionEncoding","repo":"melmaphother/timedart","repo_kind":"official","path":"models/TimeDART.py","file_url":"https://github.com/melmaphother/timedart/blob/HEAD/models/TimeDART.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"d8cfebf163c1dda7"}},{"code_sha256_prefix":"7f6d8982e2917db0","entry":"AddSosTokenAndDropLast","repo":"melmaphother/timedart","repo_kind":"official","path":"models/TimeDART.py","file_url":"https://github.com/melmaphother/timedart/blob/HEAD/models/TimeDART.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"7f6d8982e2917db0"}},{"code_sha256_prefix":"3e37f2ef318e5fa9","entry":"Attention","repo":"mingyue-cheng/timemae","repo_kind":"listed","path":"model/TimeMAE.py","file_url":"https://github.com/mingyue-cheng/timemae/blob/HEAD/model/TimeMAE.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"3e37f2ef318e5fa9"}},{"code_sha256_prefix":"92cc2f589318058a","entry":"CausalTransformer","repo":"melmaphother/timedart","repo_kind":"official","path":"models/TimeDART.py","file_url":"https://github.com/melmaphother/timedart/blob/HEAD/models/TimeDART.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"92cc2f589318058a"}},{"code_sha256_prefix":"1cfd184f35698541","entry":"ChannelIndependence","repo":"melmaphother/timedart","repo_kind":"official","path":"models/TimeDART.py","file_url":"https://github.com/melmaphother/timedart/blob/HEAD/models/TimeDART.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"1cfd184f35698541"}},{"code_sha256_prefix":"a04e0182c092d385","entry":"CrossAttnTRMBlock","repo":"mingyue-cheng/timemae","repo_kind":"listed","path":"model/TimeMAE.py","file_url":"https://github.com/mingyue-cheng/timemae/blob/HEAD/model/TimeMAE.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"a04e0182c092d385"}},{"code_sha256_prefix":"66860e86df16e0f0","entry":"DenoisingPatchDecoder","repo":"melmaphother/timedart","repo_kind":"official","path":"models/TimeDART.py","file_url":"https://github.com/melmaphother/timedart/blob/HEAD/models/TimeDART.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"66860e86df16e0f0"}},{"code_sha256_prefix":"990b53fac39c8e9a","entry":"Encoder","repo":"mingyue-cheng/timemae","repo_kind":"listed","path":"model/TimeMAE.py","file_url":"https://github.com/mingyue-cheng/timemae/blob/HEAD/model/TimeMAE.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"990b53fac39c8e9a"}},{"code_sha256_prefix":"87541709fdeeeb38","entry":"FlattenHead","repo":"melmaphother/timedart","repo_kind":"official","path":"models/TimeDART.py","file_url":"https://github.com/melmaphother/timedart/blob/HEAD/models/TimeDART.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"87541709fdeeeb38"}},{"code_sha256_prefix":"c81e84e4981f1c79","entry":"MultiHeadAttention","repo":"mingyue-cheng/timemae","repo_kind":"listed","path":"model/TimeMAE.py","file_url":"https://github.com/mingyue-cheng/timemae/blob/HEAD/model/TimeMAE.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"c81e84e4981f1c79"}},{"code_sha256_prefix":"fdb13510f96817bd","entry":"Patch","repo":"melmaphother/timedart","repo_kind":"official","path":"models/TimeDART.py","file_url":"https://github.com/melmaphother/timedart/blob/HEAD/models/TimeDART.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"fdb13510f96817bd"}},{"code_sha256_prefix":"e7d26d51c61fe223","entry":"PatchEmbedding","repo":"melmaphother/timedart","repo_kind":"official","path":"models/TimeDART.py","file_url":"https://github.com/melmaphother/timedart/blob/HEAD/models/TimeDART.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"e7d26d51c61fe223"}},{"code_sha256_prefix":"955e820994086900","entry":"PointWiseFeedForward","repo":"mingyue-cheng/timemae","repo_kind":"listed","path":"model/TimeMAE.py","file_url":"https://github.com/mingyue-cheng/timemae/blob/HEAD/model/TimeMAE.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"955e820994086900"}},{"code_sha256_prefix":"263f21894d0b5f3c","entry":"PositionalEmbedding","repo":"mingyue-cheng/timemae","repo_kind":"listed","path":"model/TimeMAE.py","file_url":"https://github.com/mingyue-cheng/timemae/blob/HEAD/model/TimeMAE.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"263f21894d0b5f3c"}},{"code_sha256_prefix":"ba3da5536ab7c680","entry":"PositionalEncoding","repo":"melmaphother/timedart","repo_kind":"official","path":"models/TimeDART.py","file_url":"https://github.com/melmaphother/timedart/blob/HEAD/models/TimeDART.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"ba3da5536ab7c680"}},{"code_sha256_prefix":"8d9d95edd7487e3f","entry":"SublayerConnection","repo":"mingyue-cheng/timemae","repo_kind":"listed","path":"model/TimeMAE.py","file_url":"https://github.com/mingyue-cheng/timemae/blob/HEAD/model/TimeMAE.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"8d9d95edd7487e3f"}},{"code_sha256_prefix":"1a49d94878ef7d04","entry":"Tokenizer","repo":"mingyue-cheng/timemae","repo_kind":"listed","path":"model/TimeMAE.py","file_url":"https://github.com/mingyue-cheng/timemae/blob/HEAD/model/TimeMAE.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"1a49d94878ef7d04"}},{"code_sha256_prefix":"d60276a4b2f78c8d","entry":"TransformerBlock","repo":"mingyue-cheng/timemae","repo_kind":"listed","path":"model/TimeMAE.py","file_url":"https://github.com/mingyue-cheng/timemae/blob/HEAD/model/TimeMAE.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"d60276a4b2f78c8d"}},{"code_sha256_prefix":"19428bb2d0882e25","entry":"TransformerDecoderBlock","repo":"melmaphother/timedart","repo_kind":"official","path":"models/TimeDART.py","file_url":"https://github.com/melmaphother/timedart/blob/HEAD/models/TimeDART.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"19428bb2d0882e25"}},{"code_sha256_prefix":"2af0df7394e58e06","entry":"conv1d_fft","repo":"Melmaphother/TimeDART","repo_kind":"official","path":"layers/ETSformer_EncDec.py","file_url":"https://github.com/Melmaphother/TimeDART/blob/HEAD/layers/ETSformer_EncDec.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"2af0df7394e58e06"}},{"code_sha256_prefix":"262630fa4aa5f0ac","entry":"generate_causal_mask","repo":"melmaphother/timedart","repo_kind":"official","path":"models/TimeDART.py","file_url":"https://github.com/melmaphother/timedart/blob/HEAD/models/TimeDART.py","link_basis":"first_harvest_node","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"262630fa4aa5f0ac"}},{"code_sha256_prefix":"270c682a67fcef02","entry":"generate_partial_mask","repo":"melmaphother/timedart","repo_kind":"official","path":"models/TimeDART.py","file_url":"https://github.com/melmaphother/timedart/blob/HEAD/models/TimeDART.py","link_basis":"first_harvest_node","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"270c682a67fcef02"}},{"code_sha256_prefix":"592ea8b254b006db","entry":"get_frequency_modes","repo":"Melmaphother/TimeDART","repo_kind":"official","path":"layers/FourierCorrelation.py","file_url":"https://github.com/Melmaphother/TimeDART/blob/HEAD/layers/FourierCorrelation.py","link_basis":"harvester_set","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"592ea8b254b006db"}},{"code_sha256_prefix":"f32036738135c8a8","entry":"get_phi_psi","repo":"Melmaphother/TimeDART","repo_kind":"official","path":"layers/MultiWaveletCorrelation.py","file_url":"https://github.com/Melmaphother/TimeDART/blob/HEAD/layers/MultiWaveletCorrelation.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"f32036738135c8a8"}},{"code_sha256_prefix":"4ef26472da51e3aa","entry":"legendreDer","repo":"Melmaphother/TimeDART","repo_kind":"official","path":"layers/MultiWaveletCorrelation.py","file_url":"https://github.com/Melmaphother/TimeDART/blob/HEAD/layers/MultiWaveletCorrelation.py","link_basis":"harvester_set","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"4ef26472da51e3aa"}},{"code_sha256_prefix":"a54c8c5c47c6a8b8","entry":"phi_","repo":"Melmaphother/TimeDART","repo_kind":"official","path":"layers/MultiWaveletCorrelation.py","file_url":"https://github.com/Melmaphother/TimeDART/blob/HEAD/layers/MultiWaveletCorrelation.py","link_basis":"harvester_set","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"a54c8c5c47c6a8b8"}},{"code_sha256_prefix":"d270abcd0d744a53","entry":"Diffusion","repo":"melmaphother/timedart","repo_kind":"official","path":"models/TimeDART.py","file_url":"https://github.com/melmaphother/timedart/blob/HEAD/models/TimeDART.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"d270abcd0d744a53"}},{"code_sha256_prefix":"92f6c7a56a9adc06","entry":"Model","repo":"melmaphother/timedart","repo_kind":"official","path":"models/TimeDART.py","file_url":"https://github.com/melmaphother/timedart/blob/HEAD/models/TimeDART.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"92f6c7a56a9adc06"}},{"code_sha256_prefix":"46aa10294e194266","entry":"Regressor","repo":"mingyue-cheng/timemae","repo_kind":"listed","path":"model/TimeMAE.py","file_url":"https://github.com/mingyue-cheng/timemae/blob/HEAD/model/TimeMAE.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"46aa10294e194266"}},{"code_sha256_prefix":"aaa32fbc007d1bdc","entry":"TimeMAE","repo":"mingyue-cheng/timemae","repo_kind":"listed","path":"model/TimeMAE.py","file_url":"https://github.com/mingyue-cheng/timemae/blob/HEAD/model/TimeMAE.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"aaa32fbc007d1bdc"}},{"code_sha256_prefix":"fee1ec512a386c64","entry":"TransformerEncoderBlock","repo":"melmaphother/timedart","repo_kind":"official","path":"models/TimeDART.py","file_url":"https://github.com/melmaphother/timedart/blob/HEAD/models/TimeDART.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"fee1ec512a386c64"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}