{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/dfferentiable-raycasting-for-self-supervised","title":"Differentiable Raycasting for Self-supervised Occupancy Forecasting","arxiv_id":"2210.01917","date":"2022-10-04","proceeding":null,"authors":["Tarasha Khurana","Peiyun Hu","Achal Dave","Jason Ziglar","David Held","Deva Ramanan"],"abstract":"Motion planning for safe autonomous driving requires learning how the environment around an ego-vehicle evolves with time. Ego-centric perception of driveable regions in a scene not only changes with the motion of actors in the environment, but also with the movement of the ego-vehicle itself. Self-supervised representations proposed for large-scale planning, such as ego-centric freespace, confound these two motions, making the representation difficult to use for downstream motion planners. In this paper, we use geometric occupancy as a natural alternative to view-dependent representations such as freespace. Occupancy maps naturally disentangle the motion of the environment from the motion of the ego-vehicle. However, one cannot directly observe the full 3D occupancy of a scene (due to occlusion), making it difficult to use as a signal for learning. Our key insight is to use differentiable raycasting to \"render\" future occupancy predictions into future LiDAR sweep predictions, which can be compared with ground-truth sweeps for self-supervised learning. The use of differentiable raycasting allows occupancy to emerge as an internal representation within the forecasting network. In the absence of groundtruth occupancy, we quantitatively evaluate the forecasting of raycasted LiDAR sweeps and show improvements of upto 15 F1 points. For downstream motion planners, where emergent occupancy can be directly used to guide non-driveable regions, this representation relatively reduces the number of collisions with objects by up to 17% as compared to freespace-centric motion planners.","url_abs":"https://arxiv.org/abs/2210.01917v2","url_pdf":"https://arxiv.org/pdf/2210.01917v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"dfferentiable-raycasting-for-self-supervised","repo_url":"https://github.com/tarashakhurana/emergent-occ-forecasting","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"autonomous-driving","task_name":"Autonomous Driving"},{"task_slug":"motion-planning","task_name":"Motion Planning"},{"task_slug":"self-supervised-learning","task_name":"Self-Supervised Learning"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2210.01917","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2210.01917"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/tarashakhurana/emergent-occ-forecasting","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":7,"unverified":13},"by_repo_kind":{"official":{"samples":20,"ran":7,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"caeaa72cca0b0383","entry":"conv3x3","repo":"tarashakhurana/emergent-occ-forecasting","repo_kind":"official","path":"model.py","file_url":"https://github.com/tarashakhurana/emergent-occ-forecasting/blob/HEAD/model.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"caeaa72cca0b0383"}},{"code_sha256_prefix":"31c0e5a46ae6999a","entry":"convex_hull_intersection","repo":"tarashakhurana/emergent-occ-forecasting","repo_kind":"official","path":"util/once_devkit/once_eval/rotate_iou_cpu_eval.py","file_url":"https://github.com/tarashakhurana/emergent-occ-forecasting/blob/HEAD/util/once_devkit/once_eval/rotate_iou_cpu_eval.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"31c0e5a46ae6999a"}},{"code_sha256_prefix":"aa8ad83ca9710526","entry":"deconv3x3","repo":"tarashakhurana/emergent-occ-forecasting","repo_kind":"official","path":"model.py","file_url":"https://github.com/tarashakhurana/emergent-occ-forecasting/blob/HEAD/model.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"aa8ad83ca9710526"}},{"code_sha256_prefix":"20fbfa2279e1ce4b","entry":"div_up","repo":"tarashakhurana/emergent-occ-forecasting","repo_kind":"official","path":"util/once_devkit/once_eval/iou_utils.py","file_url":"https://github.com/tarashakhurana/emergent-occ-forecasting/blob/HEAD/util/once_devkit/once_eval/iou_utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"20fbfa2279e1ce4b"}},{"code_sha256_prefix":"c5be4e15ce21fd81","entry":"maxpool2x2","repo":"tarashakhurana/emergent-occ-forecasting","repo_kind":"official","path":"model.py","file_url":"https://github.com/tarashakhurana/emergent-occ-forecasting/blob/HEAD/model.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c5be4e15ce21fd81"}},{"code_sha256_prefix":"e5929ca970405ef7","entry":"poly_area","repo":"tarashakhurana/emergent-occ-forecasting","repo_kind":"official","path":"util/once_devkit/once_eval/rotate_iou_cpu_eval.py","file_url":"https://github.com/tarashakhurana/emergent-occ-forecasting/blob/HEAD/util/once_devkit/once_eval/rotate_iou_cpu_eval.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e5929ca970405ef7"}},{"code_sha256_prefix":"05b20b55c5918c4a","entry":"polygon_clip","repo":"tarashakhurana/emergent-occ-forecasting","repo_kind":"official","path":"util/once_devkit/once_eval/rotate_iou_cpu_eval.py","file_url":"https://github.com/tarashakhurana/emergent-occ-forecasting/blob/HEAD/util/once_devkit/once_eval/rotate_iou_cpu_eval.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"05b20b55c5918c4a"}},{"code_sha256_prefix":"18273d36f07fdecf","entry":"CollateFn","repo":"tarashakhurana/emergent-occ-forecasting","repo_kind":"official","path":"data.py","file_url":"https://github.com/tarashakhurana/emergent-occ-forecasting/blob/HEAD/data.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"18273d36f07fdecf"}},{"code_sha256_prefix":"f7291472e2400f04","entry":"accumulate_scores","repo":"tarashakhurana/emergent-occ-forecasting","repo_kind":"official","path":"util/once_devkit/once_eval/evaluation.py","file_url":"https://github.com/tarashakhurana/emergent-occ-forecasting/blob/HEAD/util/once_devkit/once_eval/evaluation.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f7291472e2400f04"}},{"code_sha256_prefix":"b4e126077df53f1a","entry":"compute_split_parts","repo":"tarashakhurana/emergent-occ-forecasting","repo_kind":"official","path":"util/once_devkit/once_eval/eval_utils.py","file_url":"https://github.com/tarashakhurana/emergent-occ-forecasting/blob/HEAD/util/once_devkit/once_eval/eval_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b4e126077df53f1a"}},{"code_sha256_prefix":"b8b1c25361b6acd3","entry":"distance_filter","repo":"tarashakhurana/emergent-occ-forecasting","repo_kind":"official","path":"util/once_devkit/once_eval/eval_utils.py","file_url":"https://github.com/tarashakhurana/emergent-occ-forecasting/blob/HEAD/util/once_devkit/once_eval/eval_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b8b1c25361b6acd3"}},{"code_sha256_prefix":"0f466ea6d440fa49","entry":"draw_object_shadow","repo":"tarashakhurana/emergent-occ-forecasting","repo_kind":"official","path":"rasterize.py","file_url":"https://github.com/tarashakhurana/emergent-occ-forecasting/blob/HEAD/rasterize.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"0f466ea6d440fa49"}},{"code_sha256_prefix":"6d8bb2f8e90e3f0e","entry":"find_nearest","repo":"tarashakhurana/emergent-occ-forecasting","repo_kind":"official","path":"data.py","file_url":"https://github.com/tarashakhurana/emergent-occ-forecasting/blob/HEAD/data.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"6d8bb2f8e90e3f0e"}},{"code_sha256_prefix":"6bb2f76b61f0c130","entry":"get_thresholds","repo":"tarashakhurana/emergent-occ-forecasting","repo_kind":"official","path":"util/once_devkit/once_eval/evaluation.py","file_url":"https://github.com/tarashakhurana/emergent-occ-forecasting/blob/HEAD/util/once_devkit/once_eval/evaluation.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"6bb2f76b61f0c130"}},{"code_sha256_prefix":"aea292196d7c9068","entry":"locate_message","repo":"tarashakhurana/emergent-occ-forecasting","repo_kind":"official","path":"data.py","file_url":"https://github.com/tarashakhurana/emergent-occ-forecasting/blob/HEAD/data.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"aea292196d7c9068"}},{"code_sha256_prefix":"978da9312c343fda","entry":"overall_filter","repo":"tarashakhurana/emergent-occ-forecasting","repo_kind":"official","path":"util/once_devkit/once_eval/eval_utils.py","file_url":"https://github.com/tarashakhurana/emergent-occ-forecasting/blob/HEAD/util/once_devkit/once_eval/eval_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"978da9312c343fda"}},{"code_sha256_prefix":"2f73fde7de6f64a3","entry":"sample","repo":"tarashakhurana/emergent-occ-forecasting","repo_kind":"official","path":"sampler.py","file_url":"https://github.com/tarashakhurana/emergent-occ-forecasting/blob/HEAD/sampler.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"2f73fde7de6f64a3"}},{"code_sha256_prefix":"831d5db43f83b4be","entry":"split_info_loader_helper","repo":"tarashakhurana/emergent-occ-forecasting","repo_kind":"official","path":"util/once_devkit/once.py","file_url":"https://github.com/tarashakhurana/emergent-occ-forecasting/blob/HEAD/util/once_devkit/once.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"831d5db43f83b4be"}},{"code_sha256_prefix":"cff9bbe2b2841671","entry":"traverse","repo":"tarashakhurana/emergent-occ-forecasting","repo_kind":"official","path":"rasterize.py","file_url":"https://github.com/tarashakhurana/emergent-occ-forecasting/blob/HEAD/rasterize.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"cff9bbe2b2841671"}},{"code_sha256_prefix":"f156020a9ce871dd","entry":"traverse_once","repo":"tarashakhurana/emergent-occ-forecasting","repo_kind":"official","path":"rasterize.py","file_url":"https://github.com/tarashakhurana/emergent-occ-forecasting/blob/HEAD/rasterize.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f156020a9ce871dd"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}