{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/read-watch-and-scream-sound-generation-from","title":"Read, Watch and Scream! Sound Generation from Text and Video","arxiv_id":"2407.05551","date":"2024-07-08","proceeding":null,"authors":["Yujin Jeong","Yunji Kim","Sanghyuk Chun","Jiyoung Lee"],"abstract":"Despite the impressive progress of multimodal generative models, video-to-audio generation still suffers from limited performance and limits the flexibility to prioritize sound synthesis for specific objects within the scene. Conversely, text-to-audio generation methods generate high-quality audio but pose challenges in ensuring comprehensive scene depiction and time-varying control. To tackle these challenges, we propose a novel video-and-text-to-audio generation method, called \\ours, where video serves as a conditional control for a text-to-audio generation model. Especially, our method estimates the structural information of sound (namely, energy) from the video while receiving key content cues from a user prompt. We employ a well-performing text-to-audio model to consolidate the video control, which is much more efficient for training multimodal diffusion models with massive triplet-paired (audio-video-text) data. In addition, by separating the generative components of audio, it becomes a more flexible system that allows users to freely adjust the energy, surrounding environment, and primary sound source according to their preferences. Experimental results demonstrate that our method shows superiority in terms of quality, controllability, and training efficiency. Code and demo are available at https://naver-ai.github.io/rewas.","url_abs":"https://arxiv.org/abs/2407.05551v2","url_pdf":"https://arxiv.org/pdf/2407.05551v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"read-watch-and-scream-sound-generation-from","repo_url":"https://github.com/naver-ai/rewas","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"NOASSERTION"}}],"tasks":[{"task_slug":"audio-generation","task_name":"Audio Generation"},{"task_slug":null,"task_name":"Triplet"},{"task_slug":"video-to-sound-generation","task_name":"Video-to-Sound Generation"}],"methods":[{"method_slug":"diffusion","method_name":"Diffusion"}],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/video-to-sound-generation-on-vgg-sound","task":"Video-to-Sound Generation","dataset":"VGG-Sound","model":"ReWas","rank_in_archive_order":7,"of":8,"metrics":{"FAD":"2.16","FD":"15.24"},"uses_additional_data":false}],"syntology":{"syntology_url":"https://syntology.ai/paper/2407.05551","atlas_url":"https://app.syntology.ai/?focus=2407.05551","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2407.05551"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/naver-ai/rewas","reach":{"status":"ok","spdx":"NOASSERTION"}}],"summary":{"ran_draft_wrong":5,"ran":10,"ran_violates":2,"ran_fixture":1,"unverified":4},"by_repo_kind":{"official":{"samples":22,"ran":18,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":22,"samples":[{"code_sha256_prefix":"9fcdaa6e423e8aa7","entry":"Normalize","repo":"naver-ai/rewas","repo_kind":"official","path":"audioldm/diffusionmodules/model.py","file_url":"https://github.com/naver-ai/rewas/blob/HEAD/audioldm/diffusionmodules/model.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"9fcdaa6e423e8aa7"}},{"code_sha256_prefix":"70427d38625cc3ad","entry":"add_audio_path","repo":"naver-ai/rewas","repo_kind":"official","path":"evaluation/clap_score.py","file_url":"https://github.com/naver-ai/rewas/blob/HEAD/evaluation/clap_score.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"70427d38625cc3ad"}},{"code_sha256_prefix":"12c09a1a10ca585d","entry":"collate_fn","repo":"naver-ai/rewas","repo_kind":"official","path":"eval_MAE.py","file_url":"https://github.com/naver-ai/rewas/blob/HEAD/eval_MAE.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"12c09a1a10ca585d"}},{"code_sha256_prefix":"424012cb37b31172","entry":"default","repo":"naver-ai/rewas","repo_kind":"official","path":"audioldm/diffusionmodules/attention.py","file_url":"https://github.com/naver-ai/rewas/blob/HEAD/audioldm/diffusionmodules/attention.py","link_basis":"harvester_set","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"424012cb37b31172"}},{"code_sha256_prefix":"6b4cfb91987e9535","entry":"default_audioldm_config","repo":"naver-ai/rewas","repo_kind":"official","path":"utils.py","file_url":"https://github.com/naver-ai/rewas/blob/HEAD/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"6b4cfb91987e9535"}},{"code_sha256_prefix":"aa5486a3650902d8","entry":"exists","repo":"naver-ai/rewas","repo_kind":"official","path":"audioldm/diffusionmodules/attention.py","file_url":"https://github.com/naver-ai/rewas/blob/HEAD/audioldm/diffusionmodules/attention.py","link_basis":"harvester_set","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"aa5486a3650902d8"}},{"code_sha256_prefix":"49a15be06f03b885","entry":"extract_frames","repo":"naver-ai/rewas","repo_kind":"official","path":"evaluation/av_align_score.py","file_url":"https://github.com/naver-ai/rewas/blob/HEAD/evaluation/av_align_score.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"49a15be06f03b885"}},{"code_sha256_prefix":"fe183cfd67adbf95","entry":"filter_common_keys","repo":"naver-ai/rewas","repo_kind":"official","path":"eval_MAE.py","file_url":"https://github.com/naver-ai/rewas/blob/HEAD/eval_MAE.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"fe183cfd67adbf95"}},{"code_sha256_prefix":"2078958ac9a616d2","entry":"get_audio","repo":"naver-ai/rewas","repo_kind":"official","path":"eval_MAE.py","file_url":"https://github.com/naver-ai/rewas/blob/HEAD/eval_MAE.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"2078958ac9a616d2"}},{"code_sha256_prefix":"0dbf855dab1ab575","entry":"get_node_name","repo":"naver-ai/rewas","repo_kind":"official","path":"tool_add_adapter.py","file_url":"https://github.com/naver-ai/rewas/blob/HEAD/tool_add_adapter.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"0dbf855dab1ab575"}},{"code_sha256_prefix":"cb49209c125de1b4","entry":"get_timestep_embedding","repo":"naver-ai/rewas","repo_kind":"official","path":"audioldm/diffusionmodules/model.py","file_url":"https://github.com/naver-ai/rewas/blob/HEAD/audioldm/diffusionmodules/model.py","link_basis":"harvester_set","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"cb49209c125de1b4"}},{"code_sha256_prefix":"777e661635ced528","entry":"load_json","repo":"naver-ai/rewas","repo_kind":"official","path":"utils.py","file_url":"https://github.com/naver-ai/rewas/blob/HEAD/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"777e661635ced528"}},{"code_sha256_prefix":"3137073275f8c21a","entry":"nonlinearity","repo":"naver-ai/rewas","repo_kind":"official","path":"audioldm/diffusionmodules/model.py","file_url":"https://github.com/naver-ai/rewas/blob/HEAD/audioldm/diffusionmodules/model.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"3137073275f8c21a"}},{"code_sha256_prefix":"17eb0a31c91b671c","entry":"normal_kl","repo":"naver-ai/rewas","repo_kind":"official","path":"audioldm/diffusionmodules/distributions.py","file_url":"https://github.com/naver-ai/rewas/blob/HEAD/audioldm/diffusionmodules/distributions.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"17eb0a31c91b671c"}},{"code_sha256_prefix":"8a42cc3d7c42a043","entry":"read_json","repo":"naver-ai/rewas","repo_kind":"official","path":"utils.py","file_url":"https://github.com/naver-ai/rewas/blob/HEAD/utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"8a42cc3d7c42a043"}},{"code_sha256_prefix":"1e27b8c004b711e5","entry":"resize_frames","repo":"naver-ai/rewas","repo_kind":"official","path":"evaluation/av_align_score.py","file_url":"https://github.com/naver-ai/rewas/blob/HEAD/evaluation/av_align_score.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"1e27b8c004b711e5"}},{"code_sha256_prefix":"9a299fe5ae09e407","entry":"uniq","repo":"naver-ai/rewas","repo_kind":"official","path":"audioldm/diffusionmodules/attention.py","file_url":"https://github.com/naver-ai/rewas/blob/HEAD/audioldm/diffusionmodules/attention.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"9a299fe5ae09e407"}},{"code_sha256_prefix":"129b804760b3115f","entry":"zero_module","repo":"naver-ai/rewas","repo_kind":"official","path":"audioldm/diffusionmodules/nn.py","file_url":"https://github.com/naver-ai/rewas/blob/HEAD/audioldm/diffusionmodules/nn.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"129b804760b3115f"}},{"code_sha256_prefix":"ecd0fc28815b65ae","entry":"avg_pool_nd","repo":"naver-ai/rewas","repo_kind":"official","path":"audioldm/diffusionmodules/nn.py","file_url":"https://github.com/naver-ai/rewas/blob/HEAD/audioldm/diffusionmodules/nn.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"ecd0fc28815b65ae"}},{"code_sha256_prefix":"7a452ac50d0da6f3","entry":"build_tsv_from_wavs","repo":"naver-ai/rewas","repo_kind":"official","path":"evaluation/clap_score.py","file_url":"https://github.com/naver-ai/rewas/blob/HEAD/evaluation/clap_score.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"7a452ac50d0da6f3"}},{"code_sha256_prefix":"20b90779443c9b64","entry":"cal_score_by_tsv","repo":"naver-ai/rewas","repo_kind":"official","path":"evaluation/clap_score.py","file_url":"https://github.com/naver-ai/rewas/blob/HEAD/evaluation/clap_score.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"20b90779443c9b64"}},{"code_sha256_prefix":"fe4eb545bbb728e0","entry":"conv_nd","repo":"naver-ai/rewas","repo_kind":"official","path":"audioldm/diffusionmodules/nn.py","file_url":"https://github.com/naver-ai/rewas/blob/HEAD/audioldm/diffusionmodules/nn.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"fe4eb545bbb728e0"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}