{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/endochat-grounded-multimodal-large-language","title":"EndoChat: Grounded Multimodal Large Language Model for Endoscopic Surgery","arxiv_id":"2501.11347","date":"2025-01-20","proceeding":null,"authors":["Guankun Wang","Long Bai","Junyi Wang","Kun Yuan","Zhen Li","Tianxu Jiang","Xiting He","Jinlin Wu","Zhen Chen","Zhen Lei","Hongbin Liu","Jiazheng Wang","Fan Zhang","Nicolas Padoy","Nassir Navab","Hongliang Ren"],"abstract":"Recently, Multimodal Large Language Models (MLLMs) have demonstrated their immense potential in computer-aided diagnosis and decision-making. In the context of robotic-assisted surgery, MLLMs can serve as effective tools for surgical training and guidance. However, there is still a lack of MLLMs specialized for surgical scene understanding in clinical applications. In this work, we introduce EndoChat to address various dialogue paradigms and subtasks in surgical scene understanding that surgeons encounter. To train our EndoChat, we construct the Surg-396K dataset through a novel pipeline that systematically extracts surgical information and generates structured annotations based on collected large-scale endoscopic surgery datasets. Furthermore, we introduce a multi-scale visual token interaction mechanism and a visual contrast-based reasoning mechanism to enhance the model's representation learning and reasoning capabilities. Our model achieves state-of-the-art performance across five dialogue paradigms and eight surgical scene understanding tasks. Additionally, we conduct evaluations with professional surgeons, most of whom provide positive feedback on collaborating with EndoChat. Overall, these results demonstrate that our EndoChat has great potential to significantly advance training and automation in robotic-assisted surgery.","url_abs":"https://arxiv.org/abs/2501.11347v1","url_pdf":"https://arxiv.org/pdf/2501.11347v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"endochat-grounded-multimodal-large-language","repo_url":"https://github.com/gkw0010/endochat","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"language-modeling","task_name":"Language Modeling"},{"task_slug":"language-modelling","task_name":"Language Modelling"},{"task_slug":"large-language-model","task_name":"Large Language Model"},{"task_slug":"multimodal-large-language-model","task_name":"Multimodal Large Language Model"},{"task_slug":"representation-learning","task_name":"Representation Learning"},{"task_slug":"scene-understanding","task_name":"Scene Understanding"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2501.11347","atlas_url":"https://app.syntology.ai/?focus=2501.11347","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2501.11347"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/gkw0010/endochat","reach":null}],"summary":{"ran_draft_wrong":3},"by_repo_kind":{"official":{"samples":3,"ran":3,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":3,"samples":[{"code_sha256_prefix":"15f1f93594928329","entry":"calc_metric_single_phrase","repo":"gkw0010/endochat","repo_kind":"official","path":"accessory/get_evaluation_result.py","file_url":"https://github.com/gkw0010/endochat/blob/HEAD/accessory/get_evaluation_result.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"15f1f93594928329"}},{"code_sha256_prefix":"1826f67bf3ac59eb","entry":"metrics_acc_fscore","repo":"gkw0010/endochat","repo_kind":"official","path":"accessory/get_evaluation_result.py","file_url":"https://github.com/gkw0010/endochat/blob/HEAD/accessory/get_evaluation_result.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"1826f67bf3ac59eb"}},{"code_sha256_prefix":"4cbf335103f9ed8b","entry":"parse_bbox_string","repo":"gkw0010/endochat","repo_kind":"official","path":"accessory/get_evaluation_result.py","file_url":"https://github.com/gkw0010/endochat/blob/HEAD/accessory/get_evaluation_result.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"4cbf335103f9ed8b"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}