{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/adapting-pre-trained-vision-models-for-novel","title":"Adapting Pre-Trained Vision Models for Novel Instance Detection and Segmentation","arxiv_id":"2405.17859","date":"2024-05-28","proceeding":null,"authors":["Yangxiao Lu","Jishnu Jaykumar P","Yunhui Guo","Nicholas Ruozzi","Yu Xiang"],"abstract":"Novel Instance Detection and Segmentation (NIDS) aims at detecting and segmenting novel object instances given a few examples of each instance. We propose a unified, simple, yet effective framework (NIDS-Net) comprising object proposal generation, embedding creation for both instance templates and proposal regions, and embedding matching for instance label assignment. Leveraging recent advancements in large vision methods, we utilize Grounding DINO and Segment Anything Model (SAM) to obtain object proposals with accurate bounding boxes and masks. Central to our approach is the generation of high-quality instance embeddings. We utilized foreground feature averages of patch embeddings from the DINOv2 ViT backbone, followed by refinement through a weight adapter mechanism that we introduce. We show experimentally that our weight adapter can adjust the embeddings locally within their feature space and effectively limit overfitting in the few-shot setting. Furthermore, the weight adapter optimizes weights to enhance the distinctiveness of instance embeddings during similarity computation. This methodology enables a straightforward matching strategy that results in significant performance gains. Our framework surpasses current state-of-the-art methods, demonstrating notable improvements in four detection datasets. In the segmentation tasks on seven core datasets of the BOP challenge, our method outperforms the leading published RGB methods and remains competitive with the best RGB-D method. We have also verified our method using real-world images from a Fetch robot and a RealSense camera. Project Page: https://irvlutd.github.io/NIDSNet/","url_abs":"https://arxiv.org/abs/2405.17859v3","url_pdf":"https://arxiv.org/pdf/2405.17859v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"adapting-pre-trained-vision-models-for-novel","repo_url":"https://github.com/youngsean/nids-net","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"instance-segmentation","task_name":"Instance Segmentation"},{"task_slug":"object-proposal-generation","task_name":"Object Proposal Generation"},{"task_slug":"semantic-segmentation","task_name":"Semantic Segmentation"}],"methods":[{"method_slug":"adapter","method_name":"Adapter"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"dino","method_name":"DINO"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"vision-transformer","method_name":"Vision Transformer"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2405.17859","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2405.17859"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/youngsean/nids-net","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":7},"by_repo_kind":{"official":{"samples":7,"ran":7,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"7f071ad395866cc1","entry":"apply_nms_to_results","repo":"youngsean/nids-net","repo_kind":"official","path":"ros/nids_net.py","file_url":"https://github.com/youngsean/nids-net/blob/HEAD/ros/nids_net.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"7f071ad395866cc1"}},{"code_sha256_prefix":"51a6654539b06a4f","entry":"find_mask_bbox","repo":"youngsean/nids-net","repo_kind":"official","path":"get_object_features_via_FFA.py","file_url":"https://github.com/youngsean/nids-net/blob/HEAD/get_object_features_via_FFA.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"51a6654539b06a4f"}},{"code_sha256_prefix":"3fa78c080af50ed0","entry":"get_args_parser","repo":"youngsean/nids-net","repo_kind":"official","path":"RoboTools_test_eval_gdino_FFA.py","file_url":"https://github.com/youngsean/nids-net/blob/HEAD/RoboTools_test_eval_gdino_FFA.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"3fa78c080af50ed0"}},{"code_sha256_prefix":"979af44dad69deec","entry":"get_args_parser","repo":"youngsean/nids-net","repo_kind":"official","path":"demo_eval_gdino_FFA.py","file_url":"https://github.com/youngsean/nids-net/blob/HEAD/demo_eval_gdino_FFA.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"979af44dad69deec"}},{"code_sha256_prefix":"5fde4e66c71686d7","entry":"get_args_parser","repo":"youngsean/nids-net","repo_kind":"official","path":"lmo_test_eval_gdino_FFA.py","file_url":"https://github.com/youngsean/nids-net/blob/HEAD/lmo_test_eval_gdino_FFA.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"5fde4e66c71686d7"}},{"code_sha256_prefix":"c56731246b8bed37","entry":"get_bounding_boxes","repo":"youngsean/nids-net","repo_kind":"official","path":"ros/nids_net.py","file_url":"https://github.com/youngsean/nids-net/blob/HEAD/ros/nids_net.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c56731246b8bed37"}},{"code_sha256_prefix":"c2b2608a6e57f95f","entry":"show_anns","repo":"youngsean/nids-net","repo_kind":"official","path":"ros/nids_net.py","file_url":"https://github.com/youngsean/nids-net/blob/HEAD/ros/nids_net.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c2b2608a6e57f95f"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}