{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/benchtemp-a-general-benchmark-for-evaluating","title":"BenchTemp: A General Benchmark for Evaluating Temporal Graph Neural Networks","arxiv_id":"2308.16385","date":"2023-08-31","proceeding":null,"authors":["Qiang Huang","Jiawei Jiang","Xi Susie Rao","Ce Zhang","Zhichao Han","Zitao Zhang","Xin Wang","Yongjun He","Quanqing Xu","Yang Zhao","Chuang Hu","Shuo Shang","Bo Du"],"abstract":"To handle graphs in which features or connectivities are evolving over time, a series of temporal graph neural networks (TGNNs) have been proposed. Despite the success of these TGNNs, the previous TGNN evaluations reveal several limitations regarding four critical issues: 1) inconsistent datasets, 2) inconsistent evaluation pipelines, 3) lacking workload diversity, and 4) lacking efficient comparison. Overall, there lacks an empirical study that puts TGNN models onto the same ground and compares them comprehensively. To this end, we propose BenchTemp, a general benchmark for evaluating TGNN models on various workloads. BenchTemp provides a set of benchmark datasets so that different TGNN models can be fairly compared. Further, BenchTemp engineers a standard pipeline that unifies the TGNN evaluation. With BenchTemp, we extensively compare the representative TGNN models on different tasks (e.g., link prediction and node classification) and settings (transductive and inductive), w.r.t. both effectiveness and efficiency metrics. We have made BenchTemp publicly available at https://github.com/qianghuangwhu/benchtemp.","url_abs":"https://arxiv.org/abs/2308.16385v1","url_pdf":"https://arxiv.org/pdf/2308.16385v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"benchtemp-a-general-benchmark-for-evaluating","repo_url":"https://github.com/qianghuangwhu/benchtemp","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"diversity","task_name":"Diversity"},{"task_slug":"link-prediction","task_name":"Link Prediction"},{"task_slug":"node-classification","task_name":"Node Classification"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2308.16385","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2308.16385"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/qianghuangwhu/benchtemp","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":3},"by_repo_kind":{"official":{"samples":3,"ran":3,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"b76ea688550cb2aa","entry":"expand_last_dim","repo":"qianghuangwhu/benchtemp","repo_kind":"official","path":"experimental_codes/cawn/module.py","file_url":"https://github.com/qianghuangwhu/benchtemp/blob/HEAD/experimental_codes/cawn/module.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b76ea688550cb2aa"}},{"code_sha256_prefix":"51b6ba533d3bf4fc","entry":"is_sorted_ascending","repo":"qianghuangwhu/benchtemp","repo_kind":"official","path":"DGraphFin/DGraphFin.py","file_url":"https://github.com/qianghuangwhu/benchtemp/blob/HEAD/DGraphFin/DGraphFin.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"51b6ba533d3bf4fc"}},{"code_sha256_prefix":"db7751379687ace1","entry":"is_sorted_descending","repo":"qianghuangwhu/benchtemp","repo_kind":"official","path":"DGraphFin/DGraphFin.py","file_url":"https://github.com/qianghuangwhu/benchtemp/blob/HEAD/DGraphFin/DGraphFin.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"db7751379687ace1"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}