{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/sounding-bodies-modeling-3d-spatial-sound-of","title":"Sounding Bodies: Modeling 3D Spatial Sound of Humans Using Body Pose and Audio","arxiv_id":"2311.06285","date":"2023-11-01","proceeding":"NeurIPS 2023 11","authors":["Xudong Xu","Dejan Markovic","Jacob Sandakly","Todd Keebler","Steven Krenn","Alexander Richard"],"abstract":"While 3D human body modeling has received much attention in computer vision, modeling the acoustic equivalent, i.e. modeling 3D spatial audio produced by body motion and speech, has fallen short in the community. To close this gap, we present a model that can generate accurate 3D spatial audio for full human bodies. The system consumes, as input, audio signals from headset microphones and body pose, and produces, as output, a 3D sound field surrounding the transmitter's body, from which spatial audio can be rendered at any arbitrary position in the 3D space. We collect a first-of-its-kind multimodal dataset of human bodies, recorded with multiple cameras and a spherical array of 345 microphones. In an empirical evaluation, we demonstrate that our model can produce accurate body-induced sound fields when trained with a suitable loss. Dataset and code are available online.","url_abs":"https://arxiv.org/abs/2311.06285v1","url_pdf":"https://arxiv.org/pdf/2311.06285v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"sounding-bodies-modeling-3d-spatial-sound-of","repo_url":"https://github.com/facebookresearch/soundingbodies","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"NOASSERTION"}}],"tasks":[{"task_slug":null,"task_name":"Position"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2311.06285","atlas_url":"https://app.syntology.ai/?focus=2311.06285","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2311.06285"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/facebookresearch/soundingbodies","reach":{"status":"ok","spdx":"NOASSERTION"}}],"summary":{"ran":2},"by_repo_kind":{"official":{"samples":2,"ran":2,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":2,"samples":[{"code_sha256_prefix":"eb71ef7bd252ccac","entry":"ane2cam","repo":"facebookresearch/soundingbodies","repo_kind":"official","path":"src/viz_utils.py","file_url":"https://github.com/facebookresearch/soundingbodies/blob/HEAD/src/viz_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"eb71ef7bd252ccac"}},{"code_sha256_prefix":"b53754ba564a4713","entry":"cam2ane","repo":"facebookresearch/soundingbodies","repo_kind":"official","path":"src/viz_utils.py","file_url":"https://github.com/facebookresearch/soundingbodies/blob/HEAD/src/viz_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"b53754ba564a4713"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}