{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/co-attention-gated-vision-language-embedding","title":"CAT-ViL: Co-Attention Gated Vision-Language Embedding for Visual Question Localized-Answering in Robotic Surgery","arxiv_id":"2307.05182","date":"2023-07-11","proceeding":null,"authors":["Long Bai","Mobarakol Islam","Hongliang Ren"],"abstract":"Medical students and junior surgeons often rely on senior surgeons and specialists to answer their questions when learning surgery. However, experts are often busy with clinical and academic work, and have little time to give guidance. Meanwhile, existing deep learning (DL)-based surgical Visual Question Answering (VQA) systems can only provide simple answers without the location of the answers. In addition, vision-language (ViL) embedding is still a less explored research in these kinds of tasks. Therefore, a surgical Visual Question Localized-Answering (VQLA) system would be helpful for medical students and junior surgeons to learn and understand from recorded surgical videos. We propose an end-to-end Transformer with the Co-Attention gaTed Vision-Language (CAT-ViL) embedding for VQLA in surgical scenarios, which does not require feature extraction through detection models. The CAT-ViL embedding module is designed to fuse multimodal features from visual and textual sources. The fused embedding will feed a standard Data-Efficient Image Transformer (DeiT) module, before the parallel classifier and detector for joint prediction. We conduct the experimental validation on public surgical videos from MICCAI EndoVis Challenge 2017 and 2018. The experimental results highlight the superior performance and robustness of our proposed model compared to the state-of-the-art approaches. Ablation studies further prove the outstanding performance of all the proposed components. The proposed method provides a promising solution for surgical scene understanding, and opens up a primary step in the Artificial Intelligence (AI)-based VQLA system for surgical training. Our code is publicly available.","url_abs":"https://arxiv.org/abs/2307.05182v3","url_pdf":"https://arxiv.org/pdf/2307.05182v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"co-attention-gated-vision-language-embedding","repo_url":"https://github.com/longbai1006/cat-vil","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"question-answering","task_name":"Question Answering"},{"task_slug":"scene-understanding","task_name":"Scene Understanding"},{"task_slug":"visual-question-answering-1","task_name":"Visual Question Answering"},{"task_slug":"visual-question-answering","task_name":"Visual Question Answering (VQA)"}],"methods":[{"method_slug":"absolute-position-encodings","method_name":"Absolute Position Encodings"},{"method_slug":"adam","method_name":"Adam"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"bpe","method_name":"BPE"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"label-smoothing","method_name":"Label Smoothing"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"position-wise-feed-forward-layer","method_name":"Position-Wise Feed-Forward Layer"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"transformer","method_name":"Transformer"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2307.05182","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2307.05182"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/longbai1006/cat-vil","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":7,"ran_draft_wrong":1},"by_repo_kind":{"official":{"samples":8,"ran":8,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"53daad1d7b99fbdf","entry":"accuracy","repo":"longbai1006/cat-vil","repo_kind":"official","path":"utils.py","file_url":"https://github.com/longbai1006/cat-vil/blob/HEAD/utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"53daad1d7b99fbdf"}},{"code_sha256_prefix":"19988d5d6c4a4d7c","entry":"calc_acc","repo":"longbai1006/cat-vil","repo_kind":"official","path":"utils.py","file_url":"https://github.com/longbai1006/cat-vil/blob/HEAD/utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"19988d5d6c4a4d7c"}},{"code_sha256_prefix":"9fd07c1d7cd49c46","entry":"calc_classwise_acc","repo":"longbai1006/cat-vil","repo_kind":"official","path":"utils.py","file_url":"https://github.com/longbai1006/cat-vil/blob/HEAD/utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"9fd07c1d7cd49c46"}},{"code_sha256_prefix":"e21bb85b8e1b4836","entry":"do_nms","repo":"longbai1006/cat-vil","repo_kind":"official","path":"dataset/feature_extraction/visual_bert/modeling_frcnn.py","file_url":"https://github.com/longbai1006/cat-vil/blob/HEAD/dataset/feature_extraction/visual_bert/modeling_frcnn.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e21bb85b8e1b4836"}},{"code_sha256_prefix":"e766a0a3063c15d0","entry":"make_mask","repo":"longbai1006/cat-vil","repo_kind":"official","path":"models/CATViLEmbedding.py","file_url":"https://github.com/longbai1006/cat-vil/blob/HEAD/models/CATViLEmbedding.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e766a0a3063c15d0"}},{"code_sha256_prefix":"cfa4887da544c990","entry":"norm_box","repo":"longbai1006/cat-vil","repo_kind":"official","path":"dataset/feature_extraction/visual_bert/modeling_frcnn.py","file_url":"https://github.com/longbai1006/cat-vil/blob/HEAD/dataset/feature_extraction/visual_bert/modeling_frcnn.py","link_basis":"plan_row","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"cfa4887da544c990"}},{"code_sha256_prefix":"54a2fff601881b32","entry":"pad_list_tensors","repo":"longbai1006/cat-vil","repo_kind":"official","path":"dataset/feature_extraction/visual_bert/modeling_frcnn.py","file_url":"https://github.com/longbai1006/cat-vil/blob/HEAD/dataset/feature_extraction/visual_bert/modeling_frcnn.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"54a2fff601881b32"}},{"code_sha256_prefix":"77586e7080d66922","entry":"tryload","repo":"longbai1006/cat-vil","repo_kind":"official","path":"dataset/feature_extraction/visual_bert/extracting_data.py","file_url":"https://github.com/longbai1006/cat-vil/blob/HEAD/dataset/feature_extraction/visual_bert/extracting_data.py","link_basis":"plan_row","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"77586e7080d66922"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}