{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/visual-in-context-prompting","title":"Visual In-Context Prompting","arxiv_id":"2311.13601","date":"2023-11-22","proceeding":"CVPR 2024 1","authors":["Feng Li","Qing Jiang","Hao Zhang","Tianhe Ren","Shilong Liu","Xueyan Zou","Huaizhe xu","Hongyang Li","Chunyuan Li","Jianwei Yang","Lei Zhang","Jianfeng Gao"],"abstract":"In-context prompting in large language models (LLMs) has become a prevalent approach to improve zero-shot capabilities, but this idea is less explored in the vision domain. Existing visual prompting methods focus on referring segmentation to segment the most relevant object, falling short of addressing many generic vision tasks like open-set segmentation and detection. In this paper, we introduce a universal visual in-context prompting framework for both tasks. In particular, we build on top of an encoder-decoder architecture, and develop a versatile prompt encoder to support a variety of prompts like strokes, boxes, and points. We further enhance it to take an arbitrary number of reference image segments as the context. Our extensive explorations show that the proposed visual in-context prompting elicits extraordinary referring and generic segmentation capabilities to refer and detect, yielding competitive performance to close-set in-domain datasets and showing promising results on many open-set segmentation datasets. By joint training on COCO and SA-1B, our model achieves $57.7$ PQ on COCO and $23.2$ PQ on ADE20K. Code will be available at https://github.com/UX-Decoder/DINOv.","url_abs":"https://arxiv.org/abs/2311.13601v1","url_pdf":"https://arxiv.org/pdf/2311.13601v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"visual-in-context-prompting","repo_url":"https://github.com/ux-decoder/dinov","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}},{"paper_slug":"visual-in-context-prompting","repo_url":"https://github.com/idea-research/t-rex","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok","spdx":"NOASSERTION"}},{"paper_slug":"visual-in-context-prompting","repo_url":"https://github.com/ux-decoder/semantic-sam","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"decoder","task_name":"Decoder"},{"task_slug":"segmentation","task_name":"Segmentation"},{"task_slug":"visual-prompting","task_name":"Visual Prompting"}],"methods":[{"method_slug":"focus","method_name":"Focus"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2311.13601","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2311.13601"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/idea-research/t-rex","reach":{"status":"ok","spdx":"NOASSERTION"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/ux-decoder/semantic-sam","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/ux-decoder/dinov","reach":{"status":"ok"}}],"summary":{"ran":3,"unverified":3},"by_repo_kind":{"official":{"samples":5,"ran":3,"repositories":1},"listed":{"samples":1,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":6,"samples":[{"code_sha256_prefix":"86c6e09f9f6f6637","entry":"is_model","repo":"ux-decoder/dinov","repo_kind":"official","path":"dinov/architectures/registry.py","file_url":"https://github.com/ux-decoder/dinov/blob/HEAD/dinov/architectures/registry.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"86c6e09f9f6f6637"}},{"code_sha256_prefix":"be2fbb8bf98a15f7","entry":"register_model","repo":"ux-decoder/dinov","repo_kind":"official","path":"dinov/architectures/registry.py","file_url":"https://github.com/ux-decoder/dinov/blob/HEAD/dinov/architectures/registry.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"be2fbb8bf98a15f7"}},{"code_sha256_prefix":"7ad1cc95b27b07cf","entry":"sem_seg_postprocess","repo":"ux-decoder/dinov","repo_kind":"official","path":"dinov/modules/postprocessing.py","file_url":"https://github.com/ux-decoder/dinov/blob/HEAD/dinov/modules/postprocessing.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"7ad1cc95b27b07cf"}},{"code_sha256_prefix":"c67e4671ab9094aa","entry":"build_model","repo":"ux-decoder/dinov","repo_kind":"official","path":"dinov/architectures/build.py","file_url":"https://github.com/ux-decoder/dinov/blob/HEAD/dinov/architectures/build.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"c67e4671ab9094aa"}},{"code_sha256_prefix":"0c3902b80e9968b5","entry":"model_entrypoints","repo":"ux-decoder/dinov","repo_kind":"official","path":"dinov/architectures/registry.py","file_url":"https://github.com/ux-decoder/dinov/blob/HEAD/dinov/architectures/registry.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"0c3902b80e9968b5"}},{"code_sha256_prefix":"31eafc8e0b8e3c1d","entry":"remove_small_regions","repo":"ux-decoder/semantic-sam","repo_kind":"listed","path":"tasks/interactive_idino_m2m.py","file_url":"https://github.com/ux-decoder/semantic-sam/blob/HEAD/tasks/interactive_idino_m2m.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"31eafc8e0b8e3c1d"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}