{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/touchstone-benchmark-are-we-on-the-right-way","title":"Touchstone Benchmark: Are We on the Right Way for Evaluating AI Algorithms for Medical Segmentation?","arxiv_id":"2411.03670","date":"2024-11-06","proceeding":null,"authors":["Pedro R. A. S. Bassi","Wenxuan Li","Yucheng Tang","Fabian Isensee","Zifu Wang","Jieneng Chen","Yu-Cheng Chou","Yannick Kirchhoff","Maximilian Rokuss","Ziyan Huang","Jin Ye","Junjun He","Tassilo Wald","Constantin Ulrich","Michael Baumgartner","Saikat Roy","Klaus H. Maier-Hein","Paul Jaeger","Yiwen Ye","Yutong Xie","Jianpeng Zhang","Ziyang Chen","Yong Xia","Zhaohu Xing","Lei Zhu","Yousef Sadegheih","Afshin Bozorgpour","Pratibha Kumari","Reza Azad","Dorit Merhof","Pengcheng Shi","Ting Ma","Yuxin Du","Fan Bai","Tiejun Huang","Bo Zhao","Haonan Wang","Xiaomeng Li","Hanxue Gu","Haoyu Dong","Jichen Yang","Maciej A. Mazurowski","Saumya Gupta","Linshan Wu","Jiaxin Zhuang","Hao Chen","Holger Roth","Daguang Xu","Matthew B. Blaschko","Sergio Decherchi","Andrea Cavalli","Alan L. Yuille","Zongwei Zhou"],"abstract":"How can we test AI performance? This question seems trivial, but it isn't. Standard benchmarks often have problems such as in-distribution and small-size test sets, oversimplified metrics, unfair comparisons, and short-term outcome pressure. As a consequence, good performance on standard benchmarks does not guarantee success in real-world scenarios. To address these problems, we present Touchstone, a large-scale collaborative segmentation benchmark of 9 types of abdominal organs. This benchmark is based on 5,195 training CT scans from 76 hospitals around the world and 5,903 testing CT scans from 11 additional hospitals. This diverse test set enhances the statistical significance of benchmark results and rigorously evaluates AI algorithms across various out-of-distribution scenarios. We invited 14 inventors of 19 AI algorithms to train their algorithms, while our team, as a third party, independently evaluated these algorithms on three test sets. In addition, we also evaluated pre-existing AI frameworks--which, differing from algorithms, are more flexible and can support different algorithms--including MONAI from NVIDIA, nnU-Net from DKFZ, and numerous other open-source frameworks. We are committed to expanding this benchmark to encourage more innovation of AI algorithms for the medical domain.","url_abs":"https://arxiv.org/abs/2411.03670v2","url_pdf":"https://arxiv.org/pdf/2411.03670v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"touchstone-benchmark-are-we-on-the-right-way","repo_url":"https://github.com/mrgiovanni/touchstone","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"none","reach":{"status":"ok","spdx":"NOASSERTION"}}],"tasks":[],"methods":[{"method_slug":"set","method_name":"SET"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":null,"atlas_url":"https://app.syntology.ai/?focus=2411.03670","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2411.03670"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/mrgiovanni/touchstone","reach":{"status":"ok","spdx":"NOASSERTION"}}],"summary":{"unverified":9},"by_repo_kind":{"official":{"samples":9,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":9,"samples":[{"code_sha256_prefix":"66ceec1fba2c0660","entry":"Kruskal_Wallis","repo":"mrgiovanni/touchstone","repo_kind":"official","path":"plot/PlotGroup.py","file_url":"https://github.com/mrgiovanni/touchstone/blob/HEAD/plot/PlotGroup.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"66ceec1fba2c0660"}},{"code_sha256_prefix":"08e9d93a795237e9","entry":"Kruskal_Wallis_Pure","repo":"mrgiovanni/touchstone","repo_kind":"official","path":"plot/PlotGroup.py","file_url":"https://github.com/mrgiovanni/touchstone/blob/HEAD/plot/PlotGroup.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"08e9d93a795237e9"}},{"code_sha256_prefix":"7d96b8ba0607e765","entry":"allign","repo":"mrgiovanni/touchstone","repo_kind":"official","path":"plot/SignificanceMaps.py","file_url":"https://github.com/mrgiovanni/touchstone/blob/HEAD/plot/SignificanceMaps.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"7d96b8ba0607e765"}},{"code_sha256_prefix":"cbfb83a6bfcbf55e","entry":"find_color","repo":"mrgiovanni/touchstone","repo_kind":"official","path":"plot/PlotGroup.py","file_url":"https://github.com/mrgiovanni/touchstone/blob/HEAD/plot/PlotGroup.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"cbfb83a6bfcbf55e"}},{"code_sha256_prefix":"115b20f533ea55b2","entry":"parse_tables","repo":"mrgiovanni/touchstone","repo_kind":"official","path":"leaderboard/build.py","file_url":"https://github.com/mrgiovanni/touchstone/blob/HEAD/leaderboard/build.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"115b20f533ea55b2"}},{"code_sha256_prefix":"347edcad909a71e8","entry":"rank","repo":"mrgiovanni/touchstone","repo_kind":"official","path":"plot/SignificanceMaps.py","file_url":"https://github.com/mrgiovanni/touchstone/blob/HEAD/plot/SignificanceMaps.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"347edcad909a71e8"}},{"code_sha256_prefix":"ca0d7e5bf1922f70","entry":"render","repo":"mrgiovanni/touchstone","repo_kind":"official","path":"leaderboard/build.py","file_url":"https://github.com/mrgiovanni/touchstone/blob/HEAD/leaderboard/build.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"ca0d7e5bf1922f70"}},{"code_sha256_prefix":"5566ce8046693ed0","entry":"strip_md","repo":"mrgiovanni/touchstone","repo_kind":"official","path":"leaderboard/build.py","file_url":"https://github.com/mrgiovanni/touchstone/blob/HEAD/leaderboard/build.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"5566ce8046693ed0"}},{"code_sha256_prefix":"7cbbf284e973d991","entry":"wilcoxon_one_sided","repo":"mrgiovanni/touchstone","repo_kind":"official","path":"plot/SignificanceMaps.py","file_url":"https://github.com/mrgiovanni/touchstone/blob/HEAD/plot/SignificanceMaps.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"7cbbf284e973d991"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}