{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/data-splits-and-metrics-for-method","title":"Data Splits and Metrics for Method Benchmarking on Surgical Action Triplet Datasets","arxiv_id":"2204.05235","date":"2022-04-11","proceeding":null,"authors":["Chinedu Innocent Nwoye","Nicolas Padoy"],"abstract":"In addition to generating data and annotations, devising sensible data splitting strategies and evaluation metrics is essential for the creation of a benchmark dataset. This practice ensures consensus on the usage of the data, homogeneous assessment, and uniform comparison of research methods on the dataset. This study focuses on CholecT50, which is a 50 video surgical dataset that formalizes surgical activities as triplets of <instrument, verb, target>. In this paper, we introduce the standard splits for the CholecT50 and CholecT45 datasets and show how they compare with existing use of the dataset. CholecT45 is the first public release of 45 videos of CholecT50 dataset. We also develop a metrics library, ivtmetrics, for model evaluation on surgical triplets. Furthermore, we conduct a benchmark study by reproducing baseline methods in the most predominantly used deep learning frameworks (PyTorch and TensorFlow) to evaluate them using the proposed data splits and metrics and release them publicly to support future research. The proposed data splits and evaluation metrics will enable global tracking of research progress on the dataset and facilitate optimal model selection for further deployment.","url_abs":"https://arxiv.org/abs/2204.05235v2","url_pdf":"https://arxiv.org/pdf/2204.05235v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"data-splits-and-metrics-for-method","repo_url":"https://github.com/camma-public/attention-tripnet","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"NOASSERTION"}},{"paper_slug":"data-splits-and-metrics-for-method","repo_url":"https://github.com/camma-public/tripnet","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"NOASSERTION"}},{"paper_slug":"data-splits-and-metrics-for-method","repo_url":"https://github.com/camma-public/ivtmetrics","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"none","reach":{"status":"ok","spdx":"BSD-2-Clause"}},{"paper_slug":"data-splits-and-metrics-for-method","repo_url":"https://github.com/CAMMA-public/cholect45","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"NOASSERTION"}},{"paper_slug":"data-splits-and-metrics-for-method","repo_url":"https://github.com/camma-public/rendezvous","is_official":0,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"NOASSERTION"}},{"paper_slug":"data-splits-and-metrics-for-method","repo_url":"https://github.com/CAMMA-public/cholect50","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"NOASSERTION"}},{"paper_slug":"data-splits-and-metrics-for-method","repo_url":"https://github.com/camma-public/mcit-ig","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"NOASSERTION"}},{"paper_slug":"data-splits-and-metrics-for-method","repo_url":"https://github.com/camma-public/rendezvous-in-time","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"action-triplet-recognition","task_name":"Action Triplet Recognition"},{"task_slug":"benchmarking","task_name":"Benchmarking"},{"task_slug":"model-selection","task_name":"Model Selection"},{"task_slug":null,"task_name":"Triplet"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/action-triplet-recognition-on-cholect45","task":"Action Triplet Recognition","dataset":"CholecT45","model":"Rendezvous","rank_in_archive_order":1,"of":3,"metrics":{"mAP":"29.4±2.8"},"uses_additional_data":false},{"leaderboard":"/sota/action-triplet-recognition-on-cholect45","task":"Action Triplet Recognition","dataset":"CholecT45","model":"Attention Tripnet","rank_in_archive_order":2,"of":3,"metrics":{"mAP":"27.2±2.7"},"uses_additional_data":false},{"leaderboard":"/sota/action-triplet-recognition-on-cholect45","task":"Action Triplet Recognition","dataset":"CholecT45","model":"Tripnet","rank_in_archive_order":3,"of":3,"metrics":{"mAP":"24.4±4.7"},"uses_additional_data":false},{"leaderboard":"/sota/action-triplet-recognition-on-cholect45-cross","task":"Action Triplet Recognition","dataset":"CholecT45 (cross-val)","model":"Rendezvous","rank_in_archive_order":3,"of":5,"metrics":{"mAP":"29.4±2.8"},"uses_additional_data":false},{"leaderboard":"/sota/action-triplet-recognition-on-cholect45-cross","task":"Action Triplet Recognition","dataset":"CholecT45 (cross-val)","model":"Attention Tripnet","rank_in_archive_order":4,"of":5,"metrics":{"mAP":"27.2±2.7"},"uses_additional_data":false},{"leaderboard":"/sota/action-triplet-recognition-on-cholect45-cross","task":"Action Triplet Recognition","dataset":"CholecT45 (cross-val)","model":"Tripnet","rank_in_archive_order":5,"of":5,"metrics":{"mAP":"24.4±4.7"},"uses_additional_data":false},{"leaderboard":"/sota/action-triplet-recognition-on-cholect50","task":"Action Triplet Recognition","dataset":"CholecT50","model":"Rendezvous (PyTorch)","rank_in_archive_order":2,"of":6,"metrics":{"Mean AP":"29.5"},"uses_additional_data":false},{"leaderboard":"/sota/action-triplet-recognition-on-cholect50","task":"Action Triplet Recognition","dataset":"CholecT50","model":"Attention Tripnet (PyTorch)","rank_in_archive_order":4,"of":6,"metrics":{"Mean AP":"23.3"},"uses_additional_data":false},{"leaderboard":"/sota/action-triplet-recognition-on-cholect50","task":"Action Triplet Recognition","dataset":"CholecT50","model":"Tripnet (PyTorch)","rank_in_archive_order":5,"of":6,"metrics":{"Mean AP":"21.6"},"uses_additional_data":false},{"leaderboard":"/sota/action-triplet-recognition-on-cholect50-1","task":"Action Triplet Recognition","dataset":"CholecT50 (Challenge)","model":"Rendezvous (PyTorch)","rank_in_archive_order":6,"of":27,"metrics":{"mAP":"32.8"},"uses_additional_data":false},{"leaderboard":"/sota/action-triplet-recognition-on-cholect50-1","task":"Action Triplet Recognition","dataset":"CholecT50 (Challenge)","model":"Attention Tripnet (PyTorch)","rank_in_archive_order":12,"of":27,"metrics":{"mAP":"27.7"},"uses_additional_data":false},{"leaderboard":"/sota/action-triplet-recognition-on-cholect50-1","task":"Action Triplet Recognition","dataset":"CholecT50 (Challenge)","model":"Tripnet (PyTorch)","rank_in_archive_order":13,"of":27,"metrics":{"mAP":"27.4"},"uses_additional_data":false},{"leaderboard":"/sota/action-triplet-recognition-on-cholect50-cross-1","task":"Action Triplet Recognition","dataset":"CholecT50 (cross-val)","model":"Rendezvous","rank_in_archive_order":2,"of":4,"metrics":{"mAP":"29.4±2.5"},"uses_additional_data":false},{"leaderboard":"/sota/action-triplet-recognition-on-cholect50-cross-1","task":"Action Triplet Recognition","dataset":"CholecT50 (cross-val)","model":"Attention Tripnet","rank_in_archive_order":3,"of":4,"metrics":{"mAP":"27.2±2.9"},"uses_additional_data":false},{"leaderboard":"/sota/action-triplet-recognition-on-cholect50-cross-1","task":"Action Triplet Recognition","dataset":"CholecT50 (cross-val)","model":"Tripnet","rank_in_archive_order":4,"of":4,"metrics":{"mAP":"25.3±2.4"},"uses_additional_data":false},{"leaderboard":"/sota/action-triplet-recognition-on-cholect50-cross","task":"Action Triplet Recognition","dataset":"CholecT50(cross-val)","model":"Rendezvous","rank_in_archive_order":1,"of":1,"metrics":{"mAP":"29.4±2.5"},"uses_additional_data":false}],"syntology":{"syntology_url":null,"atlas_url":"https://app.syntology.ai/?focus=2204.05235","mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}