{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/frieren-efficient-video-to-audio-generation","title":"Frieren: Efficient Video-to-Audio Generation Network with Rectified Flow Matching","arxiv_id":"2406.00320","date":"2024-06-01","proceeding":null,"authors":["Yongqi Wang","Wenxiang Guo","Rongjie Huang","Jiawei Huang","Zehan Wang","Fuming You","RuiQi Li","Zhou Zhao"],"abstract":"Video-to-audio (V2A) generation aims to synthesize content-matching audio from silent video, and it remains challenging to build V2A models with high generation quality, efficiency, and visual-audio temporal synchrony. We propose Frieren, a V2A model based on rectified flow matching. Frieren regresses the conditional transport vector field from noise to spectrogram latent with straight paths and conducts sampling by solving ODE, outperforming autoregressive and score-based models in terms of audio quality. By employing a non-autoregressive vector field estimator based on a feed-forward transformer and channel-level cross-modal feature fusion with strong temporal alignment, our model generates audio that is highly synchronized with the input video. Furthermore, through reflow and one-step distillation with guided vector field, our model can generate decent audio in a few, or even only one sampling step. Experiments indicate that Frieren achieves state-of-the-art performance in both generation quality and temporal alignment on VGGSound, with alignment accuracy reaching 97.22%, and 6.2% improvement in inception score over the strong diffusion-based baseline. Audio samples are available at http://frieren-v2a.github.io.","url_abs":"https://arxiv.org/abs/2406.00320v4","url_pdf":"https://arxiv.org/pdf/2406.00320v4.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"frieren-efficient-video-to-audio-generation","repo_url":"https://github.com/cyanbx/Frieren-V2A","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"audio-generation","task_name":"Audio Generation"},{"task_slug":"video-to-sound-generation","task_name":"Video-to-Sound Generation"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/video-to-sound-generation-on-vgg-sound","task":"Video-to-Sound Generation","dataset":"VGG-Sound","model":"Frieren","rank_in_archive_order":4,"of":8,"metrics":{"FAD":"1.32","FD":"12.26"},"uses_additional_data":false}],"syntology":{"syntology_url":"https://syntology.ai/paper/2406.00320","atlas_url":"https://app.syntology.ai/?focus=2406.00320","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2406.00320"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/cyanbx/Frieren-V2A","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran_draft_wrong":5,"ran_violates":3,"ran":5,"ran_honours":1,"unverified":3},"by_repo_kind":{"official":{"samples":17,"ran":14,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"9fcdaa6e423e8aa7","entry":"Normalize","repo":"cyanbx/Frieren-V2A","repo_kind":"official","path":"Frieren/cfm/models/autoencoder1d.py","file_url":"https://github.com/cyanbx/Frieren-V2A/blob/HEAD/Frieren/cfm/models/autoencoder1d.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"9fcdaa6e423e8aa7"}},{"code_sha256_prefix":"424012cb37b31172","entry":"default","repo":"cyanbx/Frieren-V2A","repo_kind":"official","path":"Frieren/cfm/modules/attention.py","file_url":"https://github.com/cyanbx/Frieren-V2A/blob/HEAD/Frieren/cfm/modules/attention.py","link_basis":"harvester_set","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"424012cb37b31172"}},{"code_sha256_prefix":"4cb732f513d69dfd","entry":"disabled_train","repo":"cyanbx/Frieren-V2A","repo_kind":"official","path":"Frieren/cfm/models/diffusion/cfm_scale_cfg.py","file_url":"https://github.com/cyanbx/Frieren-V2A/blob/HEAD/Frieren/cfm/models/diffusion/cfm_scale_cfg.py","link_basis":"harvester_set","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"4cb732f513d69dfd"}},{"code_sha256_prefix":"7137577cbef51217","entry":"discriminator_loss","repo":"cyanbx/Frieren-V2A","repo_kind":"official","path":"Frieren/vocoder/bigvgan/models.py","file_url":"https://github.com/cyanbx/Frieren-V2A/blob/HEAD/Frieren/vocoder/bigvgan/models.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"7137577cbef51217"}},{"code_sha256_prefix":"aa5486a3650902d8","entry":"exists","repo":"cyanbx/Frieren-V2A","repo_kind":"official","path":"Frieren/cfm/modules/attention.py","file_url":"https://github.com/cyanbx/Frieren-V2A/blob/HEAD/Frieren/cfm/modules/attention.py","link_basis":"harvester_set","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"aa5486a3650902d8"}},{"code_sha256_prefix":"e453b51f0ed5fb28","entry":"feature_loss","repo":"cyanbx/Frieren-V2A","repo_kind":"official","path":"Frieren/vocoder/bigvgan/models.py","file_url":"https://github.com/cyanbx/Frieren-V2A/blob/HEAD/Frieren/vocoder/bigvgan/models.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"e453b51f0ed5fb28"}},{"code_sha256_prefix":"221b2d116fdf1032","entry":"get_obj_from_str","repo":"cyanbx/Frieren-V2A","repo_kind":"official","path":"Frieren/cfm/logger_maa2.py","file_url":"https://github.com/cyanbx/Frieren-V2A/blob/HEAD/Frieren/cfm/logger_maa2.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"221b2d116fdf1032"}},{"code_sha256_prefix":"a26f85d7c72ef39a","entry":"get_padding","repo":"cyanbx/Frieren-V2A","repo_kind":"official","path":"Frieren/vocoder/bigvgan/models.py","file_url":"https://github.com/cyanbx/Frieren-V2A/blob/HEAD/Frieren/vocoder/bigvgan/models.py","link_basis":"harvester_set","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"a26f85d7c72ef39a"}},{"code_sha256_prefix":"997164aad25b0678","entry":"interpolate","repo":"cyanbx/Frieren-V2A","repo_kind":"official","path":"Frieren/cfm/models/cavp_modules.py","file_url":"https://github.com/cyanbx/Frieren-V2A/blob/HEAD/Frieren/cfm/models/cavp_modules.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"997164aad25b0678"}},{"code_sha256_prefix":"d72762b700feee6f","entry":"ismap","repo":"cyanbx/Frieren-V2A","repo_kind":"official","path":"Frieren/cfm/util.py","file_url":"https://github.com/cyanbx/Frieren-V2A/blob/HEAD/Frieren/cfm/util.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"d72762b700feee6f"}},{"code_sha256_prefix":"3137073275f8c21a","entry":"nonlinearity","repo":"cyanbx/Frieren-V2A","repo_kind":"official","path":"Frieren/cfm/models/autoencoder1d.py","file_url":"https://github.com/cyanbx/Frieren-V2A/blob/HEAD/Frieren/cfm/models/autoencoder1d.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"3137073275f8c21a"}},{"code_sha256_prefix":"17eb0a31c91b671c","entry":"normal_kl","repo":"cyanbx/Frieren-V2A","repo_kind":"official","path":"Frieren/cfm/models/distribution.py","file_url":"https://github.com/cyanbx/Frieren-V2A/blob/HEAD/Frieren/cfm/models/distribution.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"17eb0a31c91b671c"}},{"code_sha256_prefix":"d48d8354986e3b0e","entry":"uniform_on_device","repo":"cyanbx/Frieren-V2A","repo_kind":"official","path":"Frieren/cfm/models/diffusion/cfm_scale_cfg.py","file_url":"https://github.com/cyanbx/Frieren-V2A/blob/HEAD/Frieren/cfm/models/diffusion/cfm_scale_cfg.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"d48d8354986e3b0e"}},{"code_sha256_prefix":"9a299fe5ae09e407","entry":"uniq","repo":"cyanbx/Frieren-V2A","repo_kind":"official","path":"Frieren/cfm/modules/attention.py","file_url":"https://github.com/cyanbx/Frieren-V2A/blob/HEAD/Frieren/cfm/modules/attention.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"9a299fe5ae09e407"}},{"code_sha256_prefix":"bacff385a3aaa795","entry":"instantiate_from_config","repo":"cyanbx/Frieren-V2A","repo_kind":"official","path":"Frieren/cfm/logger_maa2.py","file_url":"https://github.com/cyanbx/Frieren-V2A/blob/HEAD/Frieren/cfm/logger_maa2.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"bacff385a3aaa795"}},{"code_sha256_prefix":"f9bd2e83191afad1","entry":"log_txt_as_img","repo":"cyanbx/Frieren-V2A","repo_kind":"official","path":"Frieren/cfm/util.py","file_url":"https://github.com/cyanbx/Frieren-V2A/blob/HEAD/Frieren/cfm/util.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"f9bd2e83191afad1"}},{"code_sha256_prefix":"c8d9d38b144e401f","entry":"md5_hash","repo":"cyanbx/Frieren-V2A","repo_kind":"official","path":"Frieren/cfm/util.py","file_url":"https://github.com/cyanbx/Frieren-V2A/blob/HEAD/Frieren/cfm/util.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"c8d9d38b144e401f"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}