{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/no-metric-to-rule-them-all-toward-principled","title":"No Metric to Rule Them All: Toward Principled Evaluations of Graph-Learning Datasets","arxiv_id":"2502.02379","date":"2025-02-04","proceeding":null,"authors":["Corinna Coupette","Jeremy Wayland","Emily Simons","Bastian Rieck"],"abstract":"Benchmark datasets have proved pivotal to the success of graph learning, and good benchmark datasets are crucial to guide the development of the field. Recent research has highlighted problems with graph-learning datasets and benchmarking practices -- revealing, for example, that methods which ignore the graph structure can outperform graph-based approaches on popular benchmark datasets. Such findings raise two questions: (1) What makes a good graph-learning dataset, and (2) how can we evaluate dataset quality in graph learning? Our work addresses these questions. As the classic evaluation setup uses datasets to evaluate models, it does not apply to dataset evaluation. Hence, we start from first principles. Observing that graph-learning datasets uniquely combine two modes -- the graph structure and the node features -- , we introduce RINGS, a flexible and extensible mode-perturbation framework to assess the quality of graph-learning datasets based on dataset ablations -- i.e., by quantifying differences between the original dataset and its perturbed representations. Within this framework, we propose two measures -- performance separability and mode complementarity -- as evaluation tools, each assessing, from a distinct angle, the capacity of a graph dataset to benchmark the power and efficacy of graph-learning methods. We demonstrate the utility of our framework for graph-learning dataset evaluation in an extensive set of experiments and derive actionable recommendations for improving the evaluation of graph-learning methods. Our work opens new research directions in data-centric graph learning, and it constitutes a first step toward the systematic evaluation of evaluations.","url_abs":"https://arxiv.org/abs/2502.02379v1","url_pdf":"https://arxiv.org/pdf/2502.02379v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"no-metric-to-rule-them-all-toward-principled","repo_url":"https://github.com/aidos-lab/rings","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"all","task_name":"All"},{"task_slug":"benchmarking","task_name":"Benchmarking"},{"task_slug":"graph-learning","task_name":"Graph Learning"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2502.02379","atlas_url":"https://app.syntology.ai/?focus=2502.02379","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2502.02379"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/aidos-lab/rings","reach":null}],"summary":{"ran_draft_wrong":1,"ran_violates":1,"ran_honours":1,"unverified":4},"by_repo_kind":{"official":{"samples":7,"ran":3,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"2fc01f9be0582f77","entry":"create_sample_distributions","repo":"aidos-lab/rings","repo_kind":"official","path":"examples/separability.py","file_url":"https://github.com/aidos-lab/rings/blob/HEAD/examples/separability.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"2fc01f9be0582f77"}},{"code_sha256_prefix":"b2321562c49429cf","entry":"maybe_normalize_diameter","repo":"aidos-lab/rings","repo_kind":"official","path":"rings/complementarity/functor.py","file_url":"https://github.com/aidos-lab/rings/blob/HEAD/rings/complementarity/functor.py","link_basis":"first_harvest_node","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"b2321562c49429cf"}},{"code_sha256_prefix":"5ef487176d2312d0","entry":"standard_feature_metrics","repo":"aidos-lab/rings","repo_kind":"official","path":"rings/complementarity/functor.py","file_url":"https://github.com/aidos-lab/rings/blob/HEAD/rings/complementarity/functor.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"5ef487176d2312d0"}},{"code_sha256_prefix":"2f198955a015236a","entry":"ComplementarityFunctor","repo":"aidos-lab/rings","repo_kind":"official","path":"rings/complementarity/functor.py","file_url":"https://github.com/aidos-lab/rings/blob/HEAD/rings/complementarity/functor.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"2f198955a015236a"}},{"code_sha256_prefix":"8c2ca21401e0f9e6","entry":"apply_perturbation","repo":"aidos-lab/rings","repo_kind":"official","path":"examples/complementarity.py","file_url":"https://github.com/aidos-lab/rings/blob/HEAD/examples/complementarity.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"8c2ca21401e0f9e6"}},{"code_sha256_prefix":"da8e8acb5bd11920","entry":"lift_attributes","repo":"aidos-lab/rings","repo_kind":"official","path":"rings/complementarity/functor.py","file_url":"https://github.com/aidos-lab/rings/blob/HEAD/rings/complementarity/functor.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"da8e8acb5bd11920"}},{"code_sha256_prefix":"4a9edcff913952ba","entry":"lift_graph","repo":"aidos-lab/rings","repo_kind":"official","path":"rings/complementarity/functor.py","file_url":"https://github.com/aidos-lab/rings/blob/HEAD/rings/complementarity/functor.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"4a9edcff913952ba"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}