{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/deisam-segment-anything-with-deictic","title":"DeiSAM: Segment Anything with Deictic Prompting","arxiv_id":"2402.14123","date":"2024-02-21","proceeding":null,"authors":["Hikaru Shindo","Manuel Brack","Gopika Sudhakaran","Devendra Singh Dhami","Patrick Schramowski","Kristian Kersting"],"abstract":"Large-scale, pre-trained neural networks have demonstrated strong capabilities in various tasks, including zero-shot image segmentation. To identify concrete objects in complex scenes, humans instinctively rely on deictic descriptions in natural language, i.e., referring to something depending on the context such as \"The object that is on the desk and behind the cup.\". However, deep learning approaches cannot reliably interpret such deictic representations due to their lack of reasoning capabilities in complex scenarios. To remedy this issue, we propose DeiSAM -- a combination of large pre-trained neural networks with differentiable logic reasoners -- for deictic promptable segmentation. Given a complex, textual segmentation description, DeiSAM leverages Large Language Models (LLMs) to generate first-order logic rules and performs differentiable forward reasoning on generated scene graphs. Subsequently, DeiSAM segments objects by matching them to the logically inferred image regions. As part of our evaluation, we propose the Deictic Visual Genome (DeiVG) dataset, containing paired visual input and complex, deictic textual prompts. Our empirical results demonstrate that DeiSAM is a substantial improvement over purely data-driven baselines for deictic promptable segmentation.","url_abs":"https://arxiv.org/abs/2402.14123v2","url_pdf":"https://arxiv.org/pdf/2402.14123v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"deisam-segment-anything-with-deictic","repo_url":"https://github.com/ml-research/deictic-segment-anything","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"image-segmentation","task_name":"Image Segmentation"},{"task_slug":"segmentation","task_name":"Segmentation"},{"task_slug":"semantic-segmentation","task_name":"Semantic Segmentation"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2402.14123","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2402.14123"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/ml-research/deictic-segment-anything","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":6,"unverified":2},"by_repo_kind":{"official":{"samples":8,"ran":6,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"f14bcfd521f5e41e","entry":"apply_default_color","repo":"ml-research/deictic-segment-anything","repo_kind":"official","path":"src/visualization_utils.py","file_url":"https://github.com/ml-research/deictic-segment-anything/blob/HEAD/src/visualization_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f14bcfd521f5e41e"}},{"code_sha256_prefix":"b5fa242dc847c110","entry":"handle_exceptions","repo":"ml-research/deictic-segment-anything","repo_kind":"official","path":"src/solve_deivg.py","file_url":"https://github.com/ml-research/deictic-segment-anything/blob/HEAD/src/solve_deivg.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b5fa242dc847c110"}},{"code_sha256_prefix":"0452a3ed40b3fdab","entry":"overlay_mask_on_image","repo":"ml-research/deictic-segment-anything","repo_kind":"official","path":"src/visualization_utils.py","file_url":"https://github.com/ml-research/deictic-segment-anything/blob/HEAD/src/visualization_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"0452a3ed40b3fdab"}},{"code_sha256_prefix":"e6f54a9871f7f74f","entry":"to_object_ids","repo":"ml-research/deictic-segment-anything","repo_kind":"official","path":"src/sam_utils.py","file_url":"https://github.com/ml-research/deictic-segment-anything/blob/HEAD/src/sam_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e6f54a9871f7f74f"}},{"code_sha256_prefix":"78373402b9704482","entry":"to_xyxy","repo":"ml-research/deictic-segment-anything","repo_kind":"official","path":"src/sam_utils.py","file_url":"https://github.com/ml-research/deictic-segment-anything/blob/HEAD/src/sam_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"78373402b9704482"}},{"code_sha256_prefix":"658c8a4cbb0a47d7","entry":"transform_boxes","repo":"ml-research/deictic-segment-anything","repo_kind":"official","path":"src/sam_utils.py","file_url":"https://github.com/ml-research/deictic-segment-anything/blob/HEAD/src/sam_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"658c8a4cbb0a47d7"}},{"code_sha256_prefix":"3f36036dd3c67eda","entry":"get_colored_mask","repo":"ml-research/deictic-segment-anything","repo_kind":"official","path":"src/visualization_utils.py","file_url":"https://github.com/ml-research/deictic-segment-anything/blob/HEAD/src/visualization_utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"3f36036dd3c67eda"}},{"code_sha256_prefix":"7f80a493d5a3ecfd","entry":"process_data","repo":"ml-research/deictic-segment-anything","repo_kind":"official","path":"src/solve_deivg.py","file_url":"https://github.com/ml-research/deictic-segment-anything/blob/HEAD/src/solve_deivg.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"7f80a493d5a3ecfd"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}