{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/nutime-numerically-multi-scaled-embedding-for","title":"NuTime: Numerically Multi-Scaled Embedding for Large-Scale Time-Series Pretraining","arxiv_id":"2310.07402","date":"2023-10-11","proceeding":null,"authors":["Chenguo Lin","Xumeng Wen","Wei Cao","Congrui Huang","Jiang Bian","Stephen Lin","Zhirong Wu"],"abstract":"Recent research on time-series self-supervised models shows great promise in learning semantic representations. However, it has been limited to small-scale datasets, e.g., thousands of temporal sequences. In this work, we make key technical contributions that are tailored to the numerical properties of time-series data and allow the model to scale to large datasets, e.g., millions of temporal sequences. We adopt the Transformer architecture by first partitioning the input into non-overlapping windows. Each window is then characterized by its normalized shape and two scalar values denoting the mean and standard deviation within each window. To embed scalar values that may possess arbitrary numerical amplitudes in a high-dimensional space, we propose a numerically multi-scaled embedding module enumerating all possible numerical scales for the scalars. The model undergoes pretraining with a simple contrastive objective on a large-scale dataset over a million sequences collected by merging existing public data. We study its transfer performance on a number of univariate and multivariate classification tasks, few shot learning, unsupervised clustering and anomaly detection benchmarks. Our method exhibits remarkable improvement against previous pretraining approaches and establishes the new state of the art, even compared with domain-specific non-learning-based methods. Code is available at: \\url{https://github.com/chenguolin/NuTime}.","url_abs":"https://arxiv.org/abs/2310.07402v3","url_pdf":"https://arxiv.org/pdf/2310.07402v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"nutime-numerically-multi-scaled-embedding-for","repo_url":"https://github.com/chenguolin/nutime","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"anomaly-detection","task_name":"Anomaly Detection"},{"task_slug":"few-shot-learning","task_name":"Few-Shot Learning"},{"task_slug":"learning-semantic-representations","task_name":"Learning Semantic Representations"},{"task_slug":"representation-learning","task_name":"Representation Learning"},{"task_slug":"temporal-sequences","task_name":"Temporal Sequences"},{"task_slug":"time-series-1","task_name":"Time Series"}],"methods":[{"method_slug":"absolute-position-encodings","method_name":"Absolute Position Encodings"},{"method_slug":"adam","method_name":"Adam"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"bpe","method_name":"BPE"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"label-smoothing","method_name":"Label Smoothing"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"position-wise-feed-forward-layer","method_name":"Position-Wise Feed-Forward Layer"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"transformer","method_name":"Transformer"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2310.07402","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2310.07402"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/chenguolin/nutime","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":4,"ran_violates":1,"unverified":2},"by_repo_kind":{"official":{"samples":7,"ran":5,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"00bcd0f720cd6f32","entry":"concat_all_gather","repo":"chenguolin/nutime","repo_kind":"official","path":"src/utils/util.py","file_url":"https://github.com/chenguolin/nutime/blob/HEAD/src/utils/util.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"00bcd0f720cd6f32"}},{"code_sha256_prefix":"e2ff86ca79e1aa79","entry":"get_logger","repo":"chenguolin/nutime","repo_kind":"official","path":"src/utils/util.py","file_url":"https://github.com/chenguolin/nutime/blob/HEAD/src/utils/util.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e2ff86ca79e1aa79"}},{"code_sha256_prefix":"e0333b66e464d925","entry":"get_relative_position_index","repo":"chenguolin/nutime","repo_kind":"official","path":"src/models/layers.py","file_url":"https://github.com/chenguolin/nutime/blob/HEAD/src/models/layers.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e0333b66e464d925"}},{"code_sha256_prefix":"939a04619f4b83e9","entry":"save_model_architecture","repo":"chenguolin/nutime","repo_kind":"official","path":"src/utils/util.py","file_url":"https://github.com/chenguolin/nutime/blob/HEAD/src/utils/util.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"939a04619f4b83e9"}},{"code_sha256_prefix":"4e1c1ced33060261","entry":"strtobool","repo":"chenguolin/nutime","repo_kind":"official","path":"src/config.py","file_url":"https://github.com/chenguolin/nutime/blob/HEAD/src/config.py","link_basis":"plan_row","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"4e1c1ced33060261"}},{"code_sha256_prefix":"87b6a0d12ced75b4","entry":"get_model","repo":"chenguolin/nutime","repo_kind":"official","path":"src/models/build.py","file_url":"https://github.com/chenguolin/nutime/blob/HEAD/src/models/build.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"87b6a0d12ced75b4"}},{"code_sha256_prefix":"f6f6bdf18b53b426","entry":"get_ssl_model","repo":"chenguolin/nutime","repo_kind":"official","path":"src/models/build.py","file_url":"https://github.com/chenguolin/nutime/blob/HEAD/src/models/build.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f6f6bdf18b53b426"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}