{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/gear-nerf-free-viewpoint-rendering-and","title":"Gear-NeRF: Free-Viewpoint Rendering and Tracking with Motion-aware Spatio-Temporal Sampling","arxiv_id":"2406.03723","date":"2024-06-06","proceeding":"CVPR 2024 1","authors":["Xinhang Liu","Yu-Wing Tai","Chi-Keung Tang","Pedro Miraldo","Suhas Lohit","Moitreya Chatterjee"],"abstract":"Extensions of Neural Radiance Fields (NeRFs) to model dynamic scenes have enabled their near photo-realistic, free-viewpoint rendering. Although these methods have shown some potential in creating immersive experiences, two drawbacks limit their ubiquity: (i) a significant reduction in reconstruction quality when the computing budget is limited, and (ii) a lack of semantic understanding of the underlying scenes. To address these issues, we introduce Gear-NeRF, which leverages semantic information from powerful image segmentation models. Our approach presents a principled way for learning a spatio-temporal (4D) semantic embedding, based on which we introduce the concept of gears to allow for stratified modeling of dynamic regions of the scene based on the extent of their motion. Such differentiation allows us to adjust the spatio-temporal sampling resolution for each region in proportion to its motion scale, achieving more photo-realistic dynamic novel view synthesis. At the same time, almost for free, our approach enables free-viewpoint tracking of objects of interest - a functionality not yet achieved by existing NeRF-based methods. Empirical studies validate the effectiveness of our method, where we achieve state-of-the-art rendering and tracking performance on multiple challenging datasets.","url_abs":"https://arxiv.org/abs/2406.03723v1","url_pdf":"https://arxiv.org/pdf/2406.03723v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"gear-nerf-free-viewpoint-rendering-and","repo_url":"https://github.com/merlresearch/Gear-NeRF","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"image-segmentation","task_name":"Image Segmentation"},{"task_slug":"nerf","task_name":"NeRF"},{"task_slug":"novel-view-synthesis","task_name":"Novel View Synthesis"},{"task_slug":"semantic-segmentation","task_name":"Semantic Segmentation"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/novel-view-synthesis-on","task":"Novel View Synthesis","dataset":".","model":"Gear-NeRF","rank_in_archive_order":1,"of":1,"metrics":{"PSNR":"32.21"},"uses_additional_data":false}],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2406.03723","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2406.03723"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/merlresearch/Gear-NeRF","reach":null}],"summary":{"ran_draft_wrong":4,"ran":4,"ran_fixture":2,"unverified":7},"by_repo_kind":{"listed":{"samples":17,"ran":10,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":17,"samples":[{"code_sha256_prefix":"cc8911e67f5b5e20","entry":"DensityLinearRender","repo":"merlresearch/Gear-NeRF","repo_kind":"listed","path":"nlf/nets/tensorf_dynamic.py","file_url":"https://github.com/merlresearch/Gear-NeRF/blob/HEAD/nlf/nets/tensorf_dynamic.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"AGPL-3.0","inline_ok":false,"mcp_get_code":{"code_sha256":"cc8911e67f5b5e20"}},{"code_sha256_prefix":"b9c912049ac97fa9","entry":"DensityRender","repo":"merlresearch/Gear-NeRF","repo_kind":"listed","path":"nlf/nets/tensorf_dynamic.py","file_url":"https://github.com/merlresearch/Gear-NeRF/blob/HEAD/nlf/nets/tensorf_dynamic.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"AGPL-3.0","inline_ok":false,"mcp_get_code":{"code_sha256":"b9c912049ac97fa9"}},{"code_sha256_prefix":"11a282750c367ece","entry":"MLPRender_Fea","repo":"merlresearch/Gear-NeRF","repo_kind":"listed","path":"nlf/nets/tensorf_dynamic.py","file_url":"https://github.com/merlresearch/Gear-NeRF/blob/HEAD/nlf/nets/tensorf_dynamic.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"AGPL-3.0","inline_ok":false,"mcp_get_code":{"code_sha256":"11a282750c367ece"}},{"code_sha256_prefix":"c8fb66ccf4ecbf25","entry":"N_to_reso","repo":"merlresearch/Gear-NeRF","repo_kind":"listed","path":"nlf/nets/tensorf_dynamic.py","file_url":"https://github.com/merlresearch/Gear-NeRF/blob/HEAD/nlf/nets/tensorf_dynamic.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"AGPL-3.0","inline_ok":false,"mcp_get_code":{"code_sha256":"c8fb66ccf4ecbf25"}},{"code_sha256_prefix":"9931a0fe15a84cbd","entry":"alpha2weights","repo":"merlresearch/Gear-NeRF","repo_kind":"listed","path":"nlf/nets/tensorf_dynamic.py","file_url":"https://github.com/merlresearch/Gear-NeRF/blob/HEAD/nlf/nets/tensorf_dynamic.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"AGPL-3.0","inline_ok":false,"mcp_get_code":{"code_sha256":"9931a0fe15a84cbd"}},{"code_sha256_prefix":"558487cf4a6668eb","entry":"cal_n_samples","repo":"merlresearch/Gear-NeRF","repo_kind":"listed","path":"nlf/nets/tensorf_dynamic.py","file_url":"https://github.com/merlresearch/Gear-NeRF/blob/HEAD/nlf/nets/tensorf_dynamic.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"AGPL-3.0","inline_ok":false,"mcp_get_code":{"code_sha256":"558487cf4a6668eb"}},{"code_sha256_prefix":"d7fb8f2c059f056f","entry":"dot","repo":"merlresearch/Gear-NeRF","repo_kind":"listed","path":"nlf/nets/tensorf_dynamic.py","file_url":"https://github.com/merlresearch/Gear-NeRF/blob/HEAD/nlf/nets/tensorf_dynamic.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"AGPL-3.0","inline_ok":false,"mcp_get_code":{"code_sha256":"d7fb8f2c059f056f"}},{"code_sha256_prefix":"d5e9d3045cbf7799","entry":"positional_encoding","repo":"merlresearch/Gear-NeRF","repo_kind":"listed","path":"nlf/nets/tensorf_dynamic.py","file_url":"https://github.com/merlresearch/Gear-NeRF/blob/HEAD/nlf/nets/tensorf_dynamic.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"AGPL-3.0","inline_ok":false,"mcp_get_code":{"code_sha256":"d5e9d3045cbf7799"}},{"code_sha256_prefix":"d3f249454302d660","entry":"raw2alpha","repo":"merlresearch/Gear-NeRF","repo_kind":"listed","path":"nlf/nets/tensorf_dynamic.py","file_url":"https://github.com/merlresearch/Gear-NeRF/blob/HEAD/nlf/nets/tensorf_dynamic.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"AGPL-3.0","inline_ok":false,"mcp_get_code":{"code_sha256":"d3f249454302d660"}},{"code_sha256_prefix":"54d7cae83b22966d","entry":"scale_shift_color_all","repo":"merlresearch/Gear-NeRF","repo_kind":"listed","path":"nlf/nets/tensorf_dynamic.py","file_url":"https://github.com/merlresearch/Gear-NeRF/blob/HEAD/nlf/nets/tensorf_dynamic.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"AGPL-3.0","inline_ok":false,"mcp_get_code":{"code_sha256":"54d7cae83b22966d"}},{"code_sha256_prefix":"0dafc222958b3d14","entry":"AlphaGridMask","repo":"merlresearch/Gear-NeRF","repo_kind":"listed","path":"nlf/nets/tensorf_dynamic.py","file_url":"https://github.com/merlresearch/Gear-NeRF/blob/HEAD/nlf/nets/tensorf_dynamic.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"AGPL-3.0","inline_ok":false,"mcp_get_code":{"code_sha256":"0dafc222958b3d14"}},{"code_sha256_prefix":"ff3010aabd7ba0a6","entry":"DensityFourierRender","repo":"merlresearch/Gear-NeRF","repo_kind":"listed","path":"nlf/nets/tensorf_dynamic.py","file_url":"https://github.com/merlresearch/Gear-NeRF/blob/HEAD/nlf/nets/tensorf_dynamic.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"AGPL-3.0","inline_ok":false,"mcp_get_code":{"code_sha256":"ff3010aabd7ba0a6"}},{"code_sha256_prefix":"c60df874e09490db","entry":"TensorBase","repo":"merlresearch/Gear-NeRF","repo_kind":"listed","path":"nlf/nets/tensorf_dynamic.py","file_url":"https://github.com/merlresearch/Gear-NeRF/blob/HEAD/nlf/nets/tensorf_dynamic.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"AGPL-3.0","inline_ok":false,"mcp_get_code":{"code_sha256":"c60df874e09490db"}},{"code_sha256_prefix":"2630ab5b1de17fba","entry":"TensorVMKeyframeTime","repo":"merlresearch/Gear-NeRF","repo_kind":"listed","path":"nlf/nets/tensorf_dynamic.py","file_url":"https://github.com/merlresearch/Gear-NeRF/blob/HEAD/nlf/nets/tensorf_dynamic.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"AGPL-3.0","inline_ok":false,"mcp_get_code":{"code_sha256":"2630ab5b1de17fba"}},{"code_sha256_prefix":"5512fa81dec7d5a7","entry":"scale_shift_color_one","repo":"merlresearch/Gear-NeRF","repo_kind":"listed","path":"nlf/nets/tensorf_dynamic.py","file_url":"https://github.com/merlresearch/Gear-NeRF/blob/HEAD/nlf/nets/tensorf_dynamic.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"AGPL-3.0","inline_ok":false,"mcp_get_code":{"code_sha256":"5512fa81dec7d5a7"}},{"code_sha256_prefix":"679516c34faf0b57","entry":"transform_color_all","repo":"merlresearch/Gear-NeRF","repo_kind":"listed","path":"nlf/nets/tensorf_dynamic.py","file_url":"https://github.com/merlresearch/Gear-NeRF/blob/HEAD/nlf/nets/tensorf_dynamic.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"AGPL-3.0","inline_ok":false,"mcp_get_code":{"code_sha256":"679516c34faf0b57"}},{"code_sha256_prefix":"f94b001dc283f522","entry":"transform_color_one","repo":"merlresearch/Gear-NeRF","repo_kind":"listed","path":"nlf/nets/tensorf_dynamic.py","file_url":"https://github.com/merlresearch/Gear-NeRF/blob/HEAD/nlf/nets/tensorf_dynamic.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"AGPL-3.0","inline_ok":false,"mcp_get_code":{"code_sha256":"f94b001dc283f522"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}