{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/relbench-a-benchmark-for-deep-learning-on","title":"RelBench: A Benchmark for Deep Learning on Relational Databases","arxiv_id":"2407.20060","date":"2024-07-29","proceeding":null,"authors":["Joshua Robinson","Rishabh Ranjan","Weihua Hu","Kexin Huang","Jiaqi Han","Alejandro Dobles","Matthias Fey","Jan E. Lenssen","Yiwen Yuan","Zecheng Zhang","Xinwei He","Jure Leskovec"],"abstract":"We present RelBench, a public benchmark for solving predictive tasks over relational databases with graph neural networks. RelBench provides databases and tasks spanning diverse domains and scales, and is intended to be a foundational infrastructure for future research. We use RelBench to conduct the first comprehensive study of Relational Deep Learning (RDL) (Fey et al., 2024), which combines graph neural network predictive models with (deep) tabular models that extract initial entity-level representations from raw tables. End-to-end learned RDL models fully exploit the predictive signal encoded in primary-foreign key links, marking a significant shift away from the dominant paradigm of manual feature engineering combined with tabular models. To thoroughly evaluate RDL against this prior gold-standard, we conduct an in-depth user study where an experienced data scientist manually engineers features for each task. In this study, RDL learns better models whilst reducing human work needed by more than an order of magnitude. This demonstrates the power of deep learning for solving predictive tasks over relational databases, opening up many new research opportunities enabled by RelBench.","url_abs":"https://arxiv.org/abs/2407.20060v1","url_pdf":"https://arxiv.org/pdf/2407.20060v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"relbench-a-benchmark-for-deep-learning-on","repo_url":"https://github.com/snap-stanford/relbench","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"relbench-a-benchmark-for-deep-learning-on","repo_url":"https://github.com/snap-stanford/relbench-user-study","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}},{"paper_slug":"relbench-a-benchmark-for-deep-learning-on","repo_url":"https://github.com/snap-stanford/relgnn","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"deep-learning","task_name":"Deep Learning"},{"task_slug":"feature-engineering","task_name":"Feature Engineering"},{"task_slug":"graph-neural-network","task_name":"Graph Neural Network"}],"methods":[{"method_slug":"graph-neural-network","method_name":"Graph Neural Network"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2407.20060","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2407.20060"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/snap-stanford/relbench","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/snap-stanford/relbench-user-study","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/snap-stanford/relgnn","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":3},"by_repo_kind":{"official":{"samples":3,"ran":3,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":3,"samples":[{"code_sha256_prefix":"7e6b1a8d1b05ba3e","entry":"feature_summary_df","repo":"snap-stanford/relbench-user-study","repo_kind":"official","path":"utils.py","file_url":"https://github.com/snap-stanford/relbench-user-study/blob/HEAD/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"7e6b1a8d1b05ba3e"}},{"code_sha256_prefix":"c82a6a1084bdfdc8","entry":"map_preds","repo":"snap-stanford/relbench-user-study","repo_kind":"official","path":"train_gbdt.py","file_url":"https://github.com/snap-stanford/relbench-user-study/blob/HEAD/train_gbdt.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"c82a6a1084bdfdc8"}},{"code_sha256_prefix":"40816a1af20043f7","entry":"render_jinja_sql","repo":"snap-stanford/relbench-user-study","repo_kind":"official","path":"utils.py","file_url":"https://github.com/snap-stanford/relbench-user-study/blob/HEAD/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"40816a1af20043f7"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}