{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/resi-a-comprehensive-benchmark-for","title":"ReSi: A Comprehensive Benchmark for Representational Similarity Measures","arxiv_id":"2408.00531","date":"2024-08-01","proceeding":null,"authors":["Max Klabunde","Tassilo Wald","Tobias Schumacher","Klaus Maier-Hein","Markus Strohmaier","Florian Lemmerich"],"abstract":"Measuring the similarity of different representations of neural architectures is a fundamental task and an open research challenge for the machine learning community. This paper presents the first comprehensive benchmark for evaluating representational similarity measures based on well-defined groundings of similarity. The representational similarity (ReSi) benchmark consists of (i) six carefully designed tests for similarity measures, (ii) 23 similarity measures, (iii) eleven neural network architectures, and (iv) six datasets, spanning over the graph, language, and vision domains. The benchmark opens up several important avenues of research on representational similarity that enable novel explorations and applications of neural architectures. We demonstrate the utility of the ReSi benchmark by conducting experiments on various neural network architectures, real world datasets and similarity measures. All components of the benchmark are publicly available and thereby facilitate systematic reproduction and production of research results. The benchmark is extensible, future research can build on and further expand it. We believe that the ReSi benchmark can serve as a sound platform catalyzing future research that aims to systematically evaluate existing and explore novel ways of comparing representations of neural architectures.","url_abs":"https://arxiv.org/abs/2408.00531v1","url_pdf":"https://arxiv.org/pdf/2408.00531v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"resi-a-comprehensive-benchmark-for","repo_url":"https://github.com/mklabunde/resi","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok","spdx":"CC-BY-4.0"}}],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":null,"atlas_url":"https://app.syntology.ai/?focus=2408.00531","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2408.00531"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/mklabunde/resi","reach":{"status":"ok","spdx":"CC-BY-4.0"}}],"summary":{"ran":5},"by_repo_kind":{"official":{"samples":5,"ran":5,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":5,"samples":[{"code_sha256_prefix":"05059c01bda86bc9","entry":"compute_metrics","repo":"mklabunde/resi","repo_kind":"official","path":"nlp/bert_finetune.py","file_url":"https://github.com/mklabunde/resi/blob/HEAD/nlp/bert_finetune.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"CC-BY-4.0","inline_ok":false,"mcp_get_code":{"code_sha256":"05059c01bda86bc9"}},{"code_sha256_prefix":"be6b5ef0642333f0","entry":"get_grouping_dataset","repo":"mklabunde/resi","repo_kind":"official","path":"vision/_introspect_model_accuracies.py","file_url":"https://github.com/mklabunde/resi/blob/HEAD/vision/_introspect_model_accuracies.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"CC-BY-4.0","inline_ok":false,"mcp_get_code":{"code_sha256":"be6b5ef0642333f0"}},{"code_sha256_prefix":"aa6161b70c72b969","entry":"tokenize_function","repo":"mklabunde/resi","repo_kind":"official","path":"nlp/bert_finetune.py","file_url":"https://github.com/mklabunde/resi/blob/HEAD/nlp/bert_finetune.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"CC-BY-4.0","inline_ok":false,"mcp_get_code":{"code_sha256":"aa6161b70c72b969"}},{"code_sha256_prefix":"e54a5403168156e8","entry":"train_epoch","repo":"mklabunde/resi","repo_kind":"official","path":"graphs/gnn.py","file_url":"https://github.com/mklabunde/resi/blob/HEAD/graphs/gnn.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"CC-BY-4.0","inline_ok":false,"mcp_get_code":{"code_sha256":"e54a5403168156e8"}},{"code_sha256_prefix":"7a6410f7a4bc0e65","entry":"train_epoch_dropout","repo":"mklabunde/resi","repo_kind":"official","path":"graphs/gnn.py","file_url":"https://github.com/mklabunde/resi/blob/HEAD/graphs/gnn.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"CC-BY-4.0","inline_ok":false,"mcp_get_code":{"code_sha256":"7a6410f7a4bc0e65"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}