{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/towards-in-context-scene-understanding","title":"Towards In-context Scene Understanding","arxiv_id":"2306.01667","date":"2023-06-02","proceeding":"NeurIPS 2023 11","authors":[],"abstract":"In-context learning$\\unicode{x2013}$the ability to configure a model's behavior with different prompts$\\unicode{x2013}$has revolutionized the field of natural language processing, alleviating the need for task-specific models and paving the way for generalist models capable of assisting with any query. Computer vision, in contrast, has largely stayed in the former regime: specialized decoders and finetuning protocols are generally required to perform dense tasks such as semantic segmentation and depth estimation. In this work we explore a simple mechanism for in-context learning of such scene understanding tasks: nearest neighbor retrieval from a prompt of annotated features. We propose a new pretraining protocol$\\unicode{x2013}$leveraging attention within and across images$\\unicode{x2013}$which yields representations particularly useful in this regime. The resulting Hummingbird model, suitably prompted, performs various scene understanding tasks without modification while approaching the performance of specialists that have been finetuned for each task. Moreover, Hummingbird can be configured to perform new tasks much more efficiently than finetuned models, raising the possibility of scene understanding in the interactive assistant regime.","url_abs":"https://arxiv.org/abs/2306.01667v2","url_pdf":"https://arxiv.org/pdf/2306.01667v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"towards-in-context-scene-understanding","repo_url":"https://github.com/vpariza/open-hummingbird-eval","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"depth-estimation","task_name":"Depth Estimation"},{"task_slug":"in-context-learning","task_name":"In-Context Learning"},{"task_slug":"retrieval","task_name":"Retrieval"},{"task_slug":"scene-understanding","task_name":"Scene Understanding"},{"task_slug":"semantic-segmentation","task_name":"Semantic Segmentation"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":null,"atlas_url":"https://app.syntology.ai/?focus=2306.01667","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2306.01667"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/vpariza/open-hummingbird-eval","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"unverified":6},"by_repo_kind":{"listed":{"samples":6,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"a92852437d910218","entry":"apply_horizontal_flip","repo":"vpariza/open-hummingbird-eval","repo_kind":"listed","path":"hbird/utils/image_transformations.py","file_url":"https://github.com/vpariza/open-hummingbird-eval/blob/HEAD/hbird/utils/image_transformations.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a92852437d910218"}},{"code_sha256_prefix":"d379a871676e8fc0","entry":"get_hbird_train_transforms_for_imgs","repo":"vpariza/open-hummingbird-eval","repo_kind":"listed","path":"hbird/utils/transforms.py","file_url":"https://github.com/vpariza/open-hummingbird-eval/blob/HEAD/hbird/utils/transforms.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d379a871676e8fc0"}},{"code_sha256_prefix":"974405d0f091a147","entry":"list_files","repo":"vpariza/open-hummingbird-eval","repo_kind":"listed","path":"hbird/utils/io.py","file_url":"https://github.com/vpariza/open-hummingbird-eval/blob/HEAD/hbird/utils/io.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"974405d0f091a147"}},{"code_sha256_prefix":"d9d0b09899aea20e","entry":"random_resize_crop","repo":"vpariza/open-hummingbird-eval","repo_kind":"listed","path":"hbird/utils/image_transformations.py","file_url":"https://github.com/vpariza/open-hummingbird-eval/blob/HEAD/hbird/utils/image_transformations.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d9d0b09899aea20e"}},{"code_sha256_prefix":"eb2df6f81672e7cf","entry":"read_file_set","repo":"vpariza/open-hummingbird-eval","repo_kind":"listed","path":"hbird/utils/io.py","file_url":"https://github.com/vpariza/open-hummingbird-eval/blob/HEAD/hbird/utils/io.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"eb2df6f81672e7cf"}},{"code_sha256_prefix":"f64df2cf48096a0c","entry":"resize","repo":"vpariza/open-hummingbird-eval","repo_kind":"listed","path":"hbird/utils/image_transformations.py","file_url":"https://github.com/vpariza/open-hummingbird-eval/blob/HEAD/hbird/utils/image_transformations.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f64df2cf48096a0c"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}