{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/segment-anything-in-3d-with-nerfs","title":"Segment Anything in 3D with Radiance Fields","arxiv_id":"2304.12308","date":"2023-04-24","proceeding":"NeurIPS 2023 11","authors":["Jiazhong Cen","Jiemin Fang","Zanwei Zhou","Chen Yang","Lingxi Xie","Xiaopeng Zhang","Wei Shen","Qi Tian"],"abstract":"The Segment Anything Model (SAM) emerges as a powerful vision foundation model to generate high-quality 2D segmentation results. This paper aims to generalize SAM to segment 3D objects. Rather than replicating the data acquisition and annotation procedure which is costly in 3D, we design an efficient solution, leveraging the radiance field as a cheap and off-the-shelf prior that connects multi-view 2D images to the 3D space. We refer to the proposed solution as SA3D, short for Segment Anything in 3D. With SA3D, the user is only required to provide a 2D segmentation prompt (e.g., rough points) for the target object in a single view, which is used to generate its corresponding 2D mask with SAM. Next, SA3D alternately performs mask inverse rendering and cross-view self-prompting across various views to iteratively refine the 3D mask of the target object. For one view, mask inverse rendering projects the 2D mask obtained by SAM into the 3D space with guidance of the density distribution learned by the radiance field for 3D mask refinement; Then, cross-view self-prompting extracts reliable prompts automatically as the input to SAM from the rendered 2D mask of the inaccurate 3D mask for a new view. We show in experiments that SA3D adapts to various scenes and achieves 3D segmentation within seconds. Our research reveals a potential methodology to lift the ability of a 2D segmentation model to 3D. Our code is available at https://github.com/Jumpat/SegmentAnythingin3D.","url_abs":"https://arxiv.org/abs/2304.12308v5","url_pdf":"https://arxiv.org/pdf/2304.12308v5.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"segment-anything-in-3d-with-nerfs","repo_url":"https://github.com/jumpat/segmentanythingin3d","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"inverse-rendering","task_name":"Inverse Rendering"},{"task_slug":"segmentation","task_name":"Segmentation"}],"methods":[{"method_slug":"sam","method_name":"SAM"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2304.12308","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2304.12308"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/Jumpat/SegmentAnythingin3D","reach":{"status":"ok","spdx":"Apache-2.0"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/jumpat/segmentanythingin3d","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran":2,"ran_draft_wrong":2,"ran_fixture":1,"unverified":14},"by_repo_kind":{"official":{"samples":19,"ran":5,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"eb2f4df4e13ab013","entry":"create_full_step_id","repo":"Jumpat/SegmentAnythingin3D","repo_kind":"official","path":"lib/dmpigo.py","file_url":"https://github.com/Jumpat/SegmentAnythingin3D/blob/HEAD/lib/dmpigo.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"eb2f4df4e13ab013"}},{"code_sha256_prefix":"049307a781780caa","entry":"get_rays","repo":"Jumpat/SegmentAnythingin3D","repo_kind":"official","path":"lib/dvgo.py","file_url":"https://github.com/Jumpat/SegmentAnythingin3D/blob/HEAD/lib/dvgo.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"049307a781780caa"}},{"code_sha256_prefix":"797ecc3ad3e96c73","entry":"get_rays_np","repo":"Jumpat/SegmentAnythingin3D","repo_kind":"official","path":"lib/dvgo.py","file_url":"https://github.com/Jumpat/SegmentAnythingin3D/blob/HEAD/lib/dvgo.py","link_basis":"harvester_set","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"797ecc3ad3e96c73"}},{"code_sha256_prefix":"0bccf0e06e6688aa","entry":"import_modules_from_strings","repo":"Jumpat/SegmentAnythingin3D","repo_kind":"official","path":"lib/config_misc.py","file_url":"https://github.com/Jumpat/SegmentAnythingin3D/blob/HEAD/lib/config_misc.py","link_basis":"plan_row","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"0bccf0e06e6688aa"}},{"code_sha256_prefix":"4366d225ac2fe16d","entry":"ndc_rays","repo":"Jumpat/SegmentAnythingin3D","repo_kind":"official","path":"lib/dvgo.py","file_url":"https://github.com/Jumpat/SegmentAnythingin3D/blob/HEAD/lib/dvgo.py","link_basis":"plan_row","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"4366d225ac2fe16d"}},{"code_sha256_prefix":"473d9d583627843c","entry":"add_args","repo":"Jumpat/SegmentAnythingin3D","repo_kind":"official","path":"lib/config_loader.py","file_url":"https://github.com/Jumpat/SegmentAnythingin3D/blob/HEAD/lib/config_loader.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"473d9d583627843c"}},{"code_sha256_prefix":"511c97b243c563d0","entry":"cal_IoU","repo":"Jumpat/SegmentAnythingin3D","repo_kind":"official","path":"lib/evaluate.py","file_url":"https://github.com/Jumpat/SegmentAnythingin3D/blob/HEAD/lib/evaluate.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"511c97b243c563d0"}},{"code_sha256_prefix":"f315b0b66ae0d0b2","entry":"cal_IoU_from_path","repo":"Jumpat/SegmentAnythingin3D","repo_kind":"official","path":"lib/evaluate.py","file_url":"https://github.com/Jumpat/SegmentAnythingin3D/blob/HEAD/lib/evaluate.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"f315b0b66ae0d0b2"}},{"code_sha256_prefix":"deea2b0cf17f617e","entry":"cal_IoU_from_paths","repo":"Jumpat/SegmentAnythingin3D","repo_kind":"official","path":"lib/evaluate.py","file_url":"https://github.com/Jumpat/SegmentAnythingin3D/blob/HEAD/lib/evaluate.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"deea2b0cf17f617e"}},{"code_sha256_prefix":"2784b7a54316720b","entry":"compute_bbox_by_cam_frustrm","repo":"Jumpat/SegmentAnythingin3D","repo_kind":"official","path":"lib/bbox_utils.py","file_url":"https://github.com/Jumpat/SegmentAnythingin3D/blob/HEAD/lib/bbox_utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"2784b7a54316720b"}},{"code_sha256_prefix":"2c392b56127c8a17","entry":"compute_bbox_by_coarse_geo","repo":"Jumpat/SegmentAnythingin3D","repo_kind":"official","path":"lib/bbox_utils.py","file_url":"https://github.com/Jumpat/SegmentAnythingin3D/blob/HEAD/lib/bbox_utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"2c392b56127c8a17"}},{"code_sha256_prefix":"67fd5c6c899ca0af","entry":"compute_tensorf_feat","repo":"Jumpat/SegmentAnythingin3D","repo_kind":"official","path":"lib/grid.py","file_url":"https://github.com/Jumpat/SegmentAnythingin3D/blob/HEAD/lib/grid.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"67fd5c6c899ca0af"}},{"code_sha256_prefix":"e6262a56833bb6d2","entry":"compute_tensorf_val","repo":"Jumpat/SegmentAnythingin3D","repo_kind":"official","path":"lib/grid.py","file_url":"https://github.com/Jumpat/SegmentAnythingin3D/blob/HEAD/lib/grid.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"e6262a56833bb6d2"}},{"code_sha256_prefix":"1c99e80d5edb7a50","entry":"fopen","repo":"Jumpat/SegmentAnythingin3D","repo_kind":"official","path":"lib/config_path.py","file_url":"https://github.com/Jumpat/SegmentAnythingin3D/blob/HEAD/lib/config_path.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"1c99e80d5edb7a50"}},{"code_sha256_prefix":"171bcb2a8ae45d10","entry":"is_filepath","repo":"Jumpat/SegmentAnythingin3D","repo_kind":"official","path":"lib/config_path.py","file_url":"https://github.com/Jumpat/SegmentAnythingin3D/blob/HEAD/lib/config_path.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"171bcb2a8ae45d10"}},{"code_sha256_prefix":"8bc5194225bbc6b4","entry":"is_str","repo":"Jumpat/SegmentAnythingin3D","repo_kind":"official","path":"lib/config_misc.py","file_url":"https://github.com/Jumpat/SegmentAnythingin3D/blob/HEAD/lib/config_misc.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"8bc5194225bbc6b4"}},{"code_sha256_prefix":"1bda25f83ef08f6f","entry":"iter_cast","repo":"Jumpat/SegmentAnythingin3D","repo_kind":"official","path":"lib/config_misc.py","file_url":"https://github.com/Jumpat/SegmentAnythingin3D/blob/HEAD/lib/config_misc.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"1bda25f83ef08f6f"}},{"code_sha256_prefix":"5b5f0ed4755e1c68","entry":"mark_image","repo":"Jumpat/SegmentAnythingin3D","repo_kind":"official","path":"lib/gui.py","file_url":"https://github.com/Jumpat/SegmentAnythingin3D/blob/HEAD/lib/gui.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"5b5f0ed4755e1c68"}},{"code_sha256_prefix":"46c1d9cb19d326b6","entry":"scandir","repo":"Jumpat/SegmentAnythingin3D","repo_kind":"official","path":"lib/config_path.py","file_url":"https://github.com/Jumpat/SegmentAnythingin3D/blob/HEAD/lib/config_path.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"46c1d9cb19d326b6"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}