{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/tfb-towards-comprehensive-and-fair","title":"TFB: Towards Comprehensive and Fair Benchmarking of Time Series Forecasting Methods","arxiv_id":"2403.20150","date":"2024-03-29","proceeding":null,"authors":["Xiangfei Qiu","Jilin Hu","Lekui Zhou","Xingjian Wu","Junyang Du","Buang Zhang","Chenjuan Guo","Aoying Zhou","Christian S. Jensen","Zhenli Sheng","Bin Yang"],"abstract":"Time series are generated in diverse domains such as economic, traffic, health, and energy, where forecasting of future values has numerous important applications. Not surprisingly, many forecasting methods are being proposed. To ensure progress, it is essential to be able to study and compare such methods empirically in a comprehensive and reliable manner. To achieve this, we propose TFB, an automated benchmark for Time Series Forecasting (TSF) methods. TFB advances the state-of-the-art by addressing shortcomings related to datasets, comparison methods, and evaluation pipelines: 1) insufficient coverage of data domains, 2) stereotype bias against traditional methods, and 3) inconsistent and inflexible pipelines. To achieve better domain coverage, we include datasets from 10 different domains: traffic, electricity, energy, the environment, nature, economic, stock markets, banking, health, and the web. We also provide a time series characterization to ensure that the selected datasets are comprehensive. To remove biases against some methods, we include a diverse range of methods, including statistical learning, machine learning, and deep learning methods, and we also support a variety of evaluation strategies and metrics to ensure a more comprehensive evaluations of different methods. To support the integration of different methods into the benchmark and enable fair comparisons, TFB features a flexible and scalable pipeline that eliminates biases. Next, we employ TFB to perform a thorough evaluation of 21 Univariate Time Series Forecasting (UTSF) methods on 8,068 univariate time series and 14 Multivariate Time Series Forecasting (MTSF) methods on 25 datasets. The benchmark code and data are available at https://github.com/decisionintelligence/TFB.","url_abs":"https://arxiv.org/abs/2403.20150v3","url_pdf":"https://arxiv.org/pdf/2403.20150v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"tfb-towards-comprehensive-and-fair","repo_url":"https://github.com/decisionintelligence/tfb","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"tfb-towards-comprehensive-and-fair","repo_url":"https://github.com/decisionintelligence/tab","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}},{"paper_slug":"tfb-towards-comprehensive-and-fair","repo_url":"https://github.com/hubtru/LTBoost","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"benchmarking","task_name":"Benchmarking"},{"task_slug":"multivariate-time-series-forecasting","task_name":"Multivariate Time Series Forecasting"},{"task_slug":"time-series-1","task_name":"Time Series"},{"task_slug":"time-series-forecasting","task_name":"Time Series Forecasting"},{"task_slug":"univariate-time-series-forecasting","task_name":"Univariate Time Series Forecasting"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2403.20150","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2403.20150"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/decisionintelligence/tfb","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/hubtru/LTBoost","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/decisionintelligence/tab","reach":{"status":"ok"}}],"summary":{"ran_draft_wrong":3,"ran_violates":1},"by_repo_kind":{"official":{"samples":4,"ran":4,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"be5cf5bae40ed05e","entry":"build_data_config","repo":"decisionintelligence/tfb","repo_kind":"official","path":"scripts/run_benchmark.py","file_url":"https://github.com/decisionintelligence/tfb/blob/HEAD/scripts/run_benchmark.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"be5cf5bae40ed05e"}},{"code_sha256_prefix":"c3ae136d000ff47d","entry":"build_model_config","repo":"decisionintelligence/tfb","repo_kind":"official","path":"scripts/run_benchmark.py","file_url":"https://github.com/decisionintelligence/tfb/blob/HEAD/scripts/run_benchmark.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c3ae136d000ff47d"}},{"code_sha256_prefix":"bd2675c612efd4a3","entry":"dropandrename_dataframe","repo":"decisionintelligence/tfb","repo_kind":"official","path":"characteristics_extractor/Characteristics_Extractor.py","file_url":"https://github.com/decisionintelligence/tfb/blob/HEAD/characteristics_extractor/Characteristics_Extractor.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"bd2675c612efd4a3"}},{"code_sha256_prefix":"06d2586e87139d81","entry":"str_to_bool","repo":"decisionintelligence/tfb","repo_kind":"official","path":"scripts/run_benchmark.py","file_url":"https://github.com/decisionintelligence/tfb/blob/HEAD/scripts/run_benchmark.py","link_basis":"first_harvest_node","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"06d2586e87139d81"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}