{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/x-drive-cross-modality-consistent-multi","title":"X-Drive: Cross-modality consistent multi-sensor data synthesis for driving scenarios","arxiv_id":"2411.01123","date":"2024-11-02","proceeding":null,"authors":["Yichen Xie","Chenfeng Xu","Chensheng Peng","Shuqi Zhao","Nhat Ho","Alexander T. Pham","Mingyu Ding","Masayoshi Tomizuka","Wei Zhan"],"abstract":"Recent advancements have exploited diffusion models for the synthesis of either LiDAR point clouds or camera image data in driving scenarios. Despite their success in modeling single-modality data marginal distribution, there is an under-exploration in the mutual reliance between different modalities to describe complex driving scenes. To fill in this gap, we propose a novel framework, X-DRIVE, to model the joint distribution of point clouds and multi-view images via a dual-branch latent diffusion model architecture. Considering the distinct geometrical spaces of the two modalities, X-DRIVE conditions the synthesis of each modality on the corresponding local regions from the other modality, ensuring better alignment and realism. To further handle the spatial ambiguity during denoising, we design the cross-modality condition module based on epipolar lines to adaptively learn the cross-modality local correspondence. Besides, X-DRIVE allows for controllable generation through multi-level input conditions, including text, bounding box, image, and point clouds. Extensive results demonstrate the high-fidelity synthetic results of X-DRIVE for both point clouds and multi-view images, adhering to input conditions while ensuring reliable cross-modality consistency. Our code will be made publicly available at https://github.com/yichen928/X-Drive.","url_abs":"https://arxiv.org/abs/2411.01123v1","url_pdf":"https://arxiv.org/pdf/2411.01123v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"x-drive-cross-modality-consistent-multi","repo_url":"https://github.com/yichen928/x-drive","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"jax","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"denoising","task_name":"Denoising"}],"methods":[{"method_slug":"diffusion","method_name":"Diffusion"},{"method_slug":"latent-diffusion-model","method_name":"Latent Diffusion Model"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2411.01123","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2411.01123"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/yichen928/x-drive","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"unverified":5},"by_repo_kind":{"official":{"samples":5,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"c397792740fe1474","entry":"get_embedder","repo":"yichen928/x-drive","repo_kind":"official","path":"xdrive/networks/embedder.py","file_url":"https://github.com/yichen928/x-drive/blob/HEAD/xdrive/networks/embedder.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"c397792740fe1474"}},{"code_sha256_prefix":"894ff15d6ddb26c0","entry":"normalizer","repo":"yichen928/x-drive","repo_kind":"official","path":"xdrive/networks/bbox_embedder.py","file_url":"https://github.com/yichen928/x-drive/blob/HEAD/xdrive/networks/bbox_embedder.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"894ff15d6ddb26c0"}},{"code_sha256_prefix":"f036b7b37ef7b81c","entry":"rotation_3d_in_axis","repo":"yichen928/x-drive","repo_kind":"official","path":"demo/helper.py","file_url":"https://github.com/yichen928/x-drive/blob/HEAD/demo/helper.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"f036b7b37ef7b81c"}},{"code_sha256_prefix":"f4e2d24b3d2bb4be","entry":"show_box_on_views","repo":"yichen928/x-drive","repo_kind":"official","path":"demo/helper.py","file_url":"https://github.com/yichen928/x-drive/blob/HEAD/demo/helper.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"f4e2d24b3d2bb4be"}},{"code_sha256_prefix":"5ac683f269170f64","entry":"visualize_camera","repo":"yichen928/x-drive","repo_kind":"official","path":"demo/helper.py","file_url":"https://github.com/yichen928/x-drive/blob/HEAD/demo/helper.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"5ac683f269170f64"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}