{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/neural-sim-learning-to-generate-training-data","title":"Neural-Sim: Learning to Generate Training Data with NeRF","arxiv_id":"2207.11368","date":"2022-07-22","proceeding":null,"authors":["Yunhao Ge","Harkirat Behl","Jiashu Xu","Suriya Gunasekar","Neel Joshi","Yale Song","Xin Wang","Laurent Itti","Vibhav Vineet"],"abstract":"Training computer vision models usually requires collecting and labeling vast amounts of imagery under a diverse set of scene configurations and properties. This process is incredibly time-consuming, and it is challenging to ensure that the captured data distribution maps well to the target domain of an application scenario. Recently, synthetic data has emerged as a way to address both of these issues. However, existing approaches either require human experts to manually tune each scene property or use automatic methods that provide little to no control; this requires rendering large amounts of random data variations, which is slow and is often suboptimal for the target domain. We present the first fully differentiable synthetic data pipeline that uses Neural Radiance Fields (NeRFs) in a closed-loop with a target application's loss function. Our approach generates data on-demand, with no human labor, to maximize accuracy for a target task. We illustrate the effectiveness of our method on synthetic and real-world object detection tasks. We also introduce a new \"YCB-in-the-Wild\" dataset and benchmark that provides a test scenario for object detection with varied poses in real-world environments.","url_abs":"https://arxiv.org/abs/2207.11368v1","url_pdf":"https://arxiv.org/pdf/2207.11368v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"neural-sim-learning-to-generate-training-data","repo_url":"https://github.com/gyhandy/neural-sim-nerf","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"nerf","task_name":"NeRF"},{"task_slug":"object-detection","task_name":"Object Detection"}],"methods":[{"method_slug":"test","method_name":"Test"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2207.11368","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2207.11368"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/gyhandy/neural-sim-nerf","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"deterministic:regex_extraction","url":"https://github.com/gyhandy/Neural-Sim-NeRF","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":4,"unverified":1},"by_repo_kind":{"official":{"samples":5,"ran":4,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"9fbeb5525d1cd232","entry":"Adam","repo":"gyhandy/Neural-Sim-NeRF","repo_kind":"official","path":"optimization/neural_sim_main.py","file_url":"https://github.com/gyhandy/Neural-Sim-NeRF/blob/HEAD/optimization/neural_sim_main.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"9fbeb5525d1cd232"}},{"code_sha256_prefix":"e71238c276b1fcdb","entry":"Momentum","repo":"gyhandy/Neural-Sim-NeRF","repo_kind":"official","path":"optimization/neural_sim_main.py","file_url":"https://github.com/gyhandy/Neural-Sim-NeRF/blob/HEAD/optimization/neural_sim_main.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e71238c276b1fcdb"}},{"code_sha256_prefix":"2428f5732ca2ea14","entry":"SGD","repo":"gyhandy/Neural-Sim-NeRF","repo_kind":"official","path":"optimization/neural_sim_main.py","file_url":"https://github.com/gyhandy/Neural-Sim-NeRF/blob/HEAD/optimization/neural_sim_main.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"2428f5732ca2ea14"}},{"code_sha256_prefix":"6821876cb729995c","entry":"adjust_learning_rate","repo":"gyhandy/Neural-Sim-NeRF","repo_kind":"official","path":"optimization/neural_sim_main.py","file_url":"https://github.com/gyhandy/Neural-Sim-NeRF/blob/HEAD/optimization/neural_sim_main.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"6821876cb729995c"}},{"code_sha256_prefix":"8132bda457768c13","entry":"bilevel_optimization","repo":"gyhandy/Neural-Sim-NeRF","repo_kind":"official","path":"optimization/neural_sim_main.py","file_url":"https://github.com/gyhandy/Neural-Sim-NeRF/blob/HEAD/optimization/neural_sim_main.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"8132bda457768c13"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}