{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/sound2vision-generating-diverse-visuals-from","title":"Sound2Vision: Generating Diverse Visuals from Audio through Cross-Modal Latent Alignment","arxiv_id":"2412.06209","date":"2024-12-09","proceeding":null,"authors":["Kim Sung-Bin","Arda Senocak","Hyunwoo Ha","Tae-Hyun Oh"],"abstract":"How does audio describe the world around us? In this work, we propose a method for generating images of visual scenes from diverse in-the-wild sounds. This cross-modal generation task is challenging due to the significant information gap between auditory and visual signals. We address this challenge by designing a model that aligns audio-visual modalities by enriching audio features with visual information and translating them into the visual latent space. These features are then fed into the pre-trained image generator to produce images. To enhance image quality, we use sound source localization to select audio-visual pairs with strong cross-modal correlations. Our method achieves substantially better results on the VEGAS and VGGSound datasets compared to previous work and demonstrates control over the generation process through simple manipulations to the input waveform or latent space. Furthermore, we analyze the geometric properties of the learned embedding space and demonstrate that our learning approach effectively aligns audio-visual signals for cross-modal generation. Based on this analysis, we show that our method is agnostic to specific design choices, showing its generalizability by integrating various model architectures and different types of audio-visual data.","url_abs":"https://arxiv.org/abs/2412.06209v1","url_pdf":"https://arxiv.org/pdf/2412.06209v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"sound2vision-generating-diverse-visuals-from","repo_url":"https://github.com/postech-ami/Sound2Scene","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"jax","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"sound-source-localization","task_name":"Sound Source Localization"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2412.06209","atlas_url":"https://app.syntology.ai/?focus=2412.06209","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2412.06209"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/postech-ami/Sound2Scene","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran_draft_wrong":5,"unverified":13},"by_repo_kind":{"official":{"samples":18,"ran":5,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"85eb95cce203dfc7","entry":"D_arch","repo":"postech-ami/Sound2Scene","repo_kind":"official","path":"BigGAN_PyTorch/BigGAN.py","file_url":"https://github.com/postech-ami/Sound2Scene/blob/HEAD/BigGAN_PyTorch/BigGAN.py","link_basis":"plan_row","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"85eb95cce203dfc7"}},{"code_sha256_prefix":"e8c308c9283407d7","entry":"G_arch","repo":"postech-ami/Sound2Scene","repo_kind":"official","path":"BigGAN_PyTorch/BigGAN.py","file_url":"https://github.com/postech-ami/Sound2Scene/blob/HEAD/BigGAN_PyTorch/BigGAN.py","link_basis":"plan_row","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e8c308c9283407d7"}},{"code_sha256_prefix":"d9def42110729a85","entry":"conv1x1","repo":"postech-ami/Sound2Scene","repo_kind":"official","path":"models/resnet.py","file_url":"https://github.com/postech-ami/Sound2Scene/blob/HEAD/models/resnet.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d9def42110729a85"}},{"code_sha256_prefix":"160bb14bd76201b4","entry":"conv3x3","repo":"postech-ami/Sound2Scene","repo_kind":"official","path":"models/resnet.py","file_url":"https://github.com/postech-ami/Sound2Scene/blob/HEAD/models/resnet.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"160bb14bd76201b4"}},{"code_sha256_prefix":"c41836f3e51fa8f1","entry":"gram_schmidt","repo":"postech-ami/Sound2Scene","repo_kind":"official","path":"BigGAN_PyTorch/layers.py","file_url":"https://github.com/postech-ami/Sound2Scene/blob/HEAD/BigGAN_PyTorch/layers.py","link_basis":"plan_row","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c41836f3e51fa8f1"}},{"code_sha256_prefix":"a99a05ac52533be9","entry":"D_arch","repo":"postech-ami/Sound2Scene","repo_kind":"official","path":"BigGAN_PyTorch/BigGANdeep.py","file_url":"https://github.com/postech-ami/Sound2Scene/blob/HEAD/BigGAN_PyTorch/BigGANdeep.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a99a05ac52533be9"}},{"code_sha256_prefix":"44d6a6343f6eb182","entry":"G_arch","repo":"postech-ami/Sound2Scene","repo_kind":"official","path":"BigGAN_PyTorch/BigGANdeep.py","file_url":"https://github.com/postech-ami/Sound2Scene/blob/HEAD/BigGAN_PyTorch/BigGANdeep.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"44d6a6343f6eb182"}},{"code_sha256_prefix":"982a26492dc6b7fb","entry":"loss_dcgan_dis","repo":"postech-ami/Sound2Scene","repo_kind":"official","path":"BigGAN_PyTorch/losses.py","file_url":"https://github.com/postech-ami/Sound2Scene/blob/HEAD/BigGAN_PyTorch/losses.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"982a26492dc6b7fb"}},{"code_sha256_prefix":"6c34831223553b6f","entry":"loss_dcgan_gen","repo":"postech-ami/Sound2Scene","repo_kind":"official","path":"BigGAN_PyTorch/losses.py","file_url":"https://github.com/postech-ami/Sound2Scene/blob/HEAD/BigGAN_PyTorch/losses.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"6c34831223553b6f"}},{"code_sha256_prefix":"290cd37ab0098d5c","entry":"loss_hinge_dis","repo":"postech-ami/Sound2Scene","repo_kind":"official","path":"BigGAN_PyTorch/losses.py","file_url":"https://github.com/postech-ami/Sound2Scene/blob/HEAD/BigGAN_PyTorch/losses.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"290cd37ab0098d5c"}},{"code_sha256_prefix":"2dda002928dada98","entry":"pnorm","repo":"postech-ami/Sound2Scene","repo_kind":"official","path":"data_utils/resnet.py","file_url":"https://github.com/postech-ami/Sound2Scene/blob/HEAD/data_utils/resnet.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"2dda002928dada98"}},{"code_sha256_prefix":"28d3a35157f2916d","entry":"power_iteration","repo":"postech-ami/Sound2Scene","repo_kind":"official","path":"BigGAN_PyTorch/layers.py","file_url":"https://github.com/postech-ami/Sound2Scene/blob/HEAD/BigGAN_PyTorch/layers.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"28d3a35157f2916d"}},{"code_sha256_prefix":"4015ab1f3ab9b881","entry":"proj","repo":"postech-ami/Sound2Scene","repo_kind":"official","path":"BigGAN_PyTorch/layers.py","file_url":"https://github.com/postech-ami/Sound2Scene/blob/HEAD/BigGAN_PyTorch/layers.py","link_basis":"plan_row","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"4015ab1f3ab9b881"}},{"code_sha256_prefix":"7ea5cc09a613ac25","entry":"resnet18","repo":"postech-ami/Sound2Scene","repo_kind":"official","path":"models/resnet.py","file_url":"https://github.com/postech-ami/Sound2Scene/blob/HEAD/models/resnet.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"7ea5cc09a613ac25"}},{"code_sha256_prefix":"49107fabd560ec05","entry":"resnet50","repo":"postech-ami/Sound2Scene","repo_kind":"official","path":"data_utils/resnet.py","file_url":"https://github.com/postech-ami/Sound2Scene/blob/HEAD/data_utils/resnet.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"49107fabd560ec05"}},{"code_sha256_prefix":"d01e23c94cc7b92c","entry":"resnext50_32x4d","repo":"postech-ami/Sound2Scene","repo_kind":"official","path":"data_utils/resnet.py","file_url":"https://github.com/postech-ami/Sound2Scene/blob/HEAD/data_utils/resnet.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d01e23c94cc7b92c"}},{"code_sha256_prefix":"98bcd15cf273e3b4","entry":"retrieve_latents","repo":"postech-ami/Sound2Scene","repo_kind":"official","path":"src/diffusers/pipelines/stable_diffusion/pipeline_stable_unclip_img2img_sound2scene.py","file_url":"https://github.com/postech-ami/Sound2Scene/blob/HEAD/src/diffusers/pipelines/stable_diffusion/pipeline_stable_unclip_img2img_sound2scene.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"98bcd15cf273e3b4"}},{"code_sha256_prefix":"8e9b27c0a20eb509","entry":"showImage","repo":"postech-ami/Sound2Scene","repo_kind":"official","path":"train_sound2vision.py","file_url":"https://github.com/postech-ami/Sound2Scene/blob/HEAD/train_sound2vision.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"8e9b27c0a20eb509"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}