{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/sonicverse-a-multisensory-simulation-platform","title":"Sonicverse: A Multisensory Simulation Platform for Embodied Household Agents that See and Hear","arxiv_id":"2306.00923","date":"2023-06-01","proceeding":null,"authors":["Ruohan Gao","Hao Li","Gokul Dharan","Zhuzhu Wang","Chengshu Li","Fei Xia","Silvio Savarese","Li Fei-Fei","Jiajun Wu"],"abstract":"Developing embodied agents in simulation has been a key research topic in recent years. Exciting new tasks, algorithms, and benchmarks have been developed in various simulators. However, most of them assume deaf agents in silent environments, while we humans perceive the world with multiple senses. We introduce Sonicverse, a multisensory simulation platform with integrated audio-visual simulation for training household agents that can both see and hear. Sonicverse models realistic continuous audio rendering in 3D environments in real-time. Together with a new audio-visual VR interface that allows humans to interact with agents with audio, Sonicverse enables a series of embodied AI tasks that need audio-visual perception. For semantic audio-visual navigation in particular, we also propose a new multi-task learning model that achieves state-of-the-art performance. In addition, we demonstrate Sonicverse's realism via sim-to-real transfer, which has not been achieved by other simulators: an agent trained in Sonicverse can successfully perform audio-visual navigation in real-world environments. Sonicverse is available at: https://github.com/StanfordVL/Sonicverse.","url_abs":"https://arxiv.org/abs/2306.00923v2","url_pdf":"https://arxiv.org/pdf/2306.00923v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"sonicverse-a-multisensory-simulation-platform","repo_url":"https://github.com/stanfordvl/sonicverse","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"none","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"multi-task-learning","task_name":"Multi-Task Learning"},{"task_slug":"visual-navigation","task_name":"Visual Navigation"}],"methods":[],"datasets_introduced":[{"slug":"sonicverse","name":"Sonicverse","full_name":""}],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2306.00923","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2306.00923"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/stanfordvl/sonicverse","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"unverified":6},"by_repo_kind":{"official":{"samples":6,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"a894eaaa1282a735","entry":"degree_order_to_index","repo":"stanfordvl/sonicverse","repo_kind":"official","path":"igibson/audio/ambisonics_example.py","file_url":"https://github.com/stanfordvl/sonicverse/blob/HEAD/igibson/audio/ambisonics_example.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a894eaaa1282a735"}},{"code_sha256_prefix":"5f1d7287a678f6d0","entry":"dumpFromRenderer","repo":"stanfordvl/sonicverse","repo_kind":"official","path":"igibson/audio/ig_acoustic_mesh.py","file_url":"https://github.com/stanfordvl/sonicverse/blob/HEAD/igibson/audio/ig_acoustic_mesh.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"5f1d7287a678f6d0"}},{"code_sha256_prefix":"50f705405369479c","entry":"index_to_degree_order","repo":"stanfordvl/sonicverse","repo_kind":"official","path":"igibson/audio/ambisonics_example.py","file_url":"https://github.com/stanfordvl/sonicverse/blob/HEAD/igibson/audio/ambisonics_example.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"50f705405369479c"}},{"code_sha256_prefix":"ab74038db1fe1e25","entry":"normalization_factor","repo":"stanfordvl/sonicverse","repo_kind":"official","path":"igibson/audio/ambisonics_example.py","file_url":"https://github.com/stanfordvl/sonicverse/blob/HEAD/igibson/audio/ambisonics_example.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ab74038db1fe1e25"}},{"code_sha256_prefix":"574f317682f05739","entry":"pixelToWorld","repo":"stanfordvl/sonicverse","repo_kind":"official","path":"igibson/audio/sound_intensity_map.py","file_url":"https://github.com/stanfordvl/sonicverse/blob/HEAD/igibson/audio/sound_intensity_map.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"574f317682f05739"}},{"code_sha256_prefix":"043e153f86e5f61b","entry":"worldToPixel","repo":"stanfordvl/sonicverse","repo_kind":"official","path":"igibson/audio/sound_intensity_map.py","file_url":"https://github.com/stanfordvl/sonicverse/blob/HEAD/igibson/audio/sound_intensity_map.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"043e153f86e5f61b"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}