{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/diff-bgm-a-diffusion-model-for-video","title":"Diff-BGM: A Diffusion Model for Video Background Music Generation","arxiv_id":"2405.11913","date":"2024-05-20","proceeding":"CVPR 2024 1","authors":["Sizhe Li","Yiming Qin","Minghang Zheng","Xin Jin","Yang Liu"],"abstract":"When editing a video, a piece of attractive background music is indispensable. However, video background music generation tasks face several challenges, for example, the lack of suitable training datasets, and the difficulties in flexibly controlling the music generation process and sequentially aligning the video and music. In this work, we first propose a high-quality music-video dataset BGM909 with detailed annotation and shot detection to provide multi-modal information about the video and music. We then present evaluation metrics to assess music quality, including music diversity and alignment between music and video with retrieval precision metrics. Finally, we propose the Diff-BGM framework to automatically generate the background music for a given video, which uses different signals to control different aspects of the music during the generation process, i.e., uses dynamic video features to control music rhythm and semantic features to control the melody and atmosphere. We propose to align the video and music sequentially by introducing a segment-aware cross-attention layer. Experiments verify the effectiveness of our proposed method. The code and models are available at https://github.com/sizhelee/Diff-BGM.","url_abs":"https://arxiv.org/abs/2405.11913v1","url_pdf":"https://arxiv.org/pdf/2405.11913v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"diff-bgm-a-diffusion-model-for-video","repo_url":"https://github.com/sizhelee/diff-bgm","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":"diversity","task_name":"Diversity"},{"task_slug":"music-generation","task_name":"Music Generation"},{"task_slug":"rhythm","task_name":"Rhythm"}],"methods":[{"method_slug":"align","method_name":"ALIGN"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2405.11913","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2405.11913"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/sizhelee/diff-bgm","reach":{"status":"ok"}},{"provenance":"deterministic:regex_extraction","url":"https://github.com/sizhelee/Diff-BGM","reach":{"status":"ok"}}],"summary":{"ran":7,"ran_draft_wrong":1,"unverified":5},"by_repo_kind":{"official":{"samples":13,"ran":8,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":13,"samples":[{"code_sha256_prefix":"ecb181d8f481c786","entry":"CrossAttention","repo":"sizhelee/Diff-BGM","repo_kind":"official","path":"diffbgm/stable_diffusion/model/unet.py","file_url":"https://github.com/sizhelee/Diff-BGM/blob/HEAD/diffbgm/stable_diffusion/model/unet.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"ecb181d8f481c786"}},{"code_sha256_prefix":"b576ab229d39e332","entry":"DownSample","repo":"sizhelee/Diff-BGM","repo_kind":"official","path":"diffbgm/stable_diffusion/model/unet.py","file_url":"https://github.com/sizhelee/Diff-BGM/blob/HEAD/diffbgm/stable_diffusion/model/unet.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"b576ab229d39e332"}},{"code_sha256_prefix":"43122f1b2e7a25bb","entry":"FeedForward","repo":"sizhelee/Diff-BGM","repo_kind":"official","path":"diffbgm/stable_diffusion/model/unet.py","file_url":"https://github.com/sizhelee/Diff-BGM/blob/HEAD/diffbgm/stable_diffusion/model/unet.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"43122f1b2e7a25bb"}},{"code_sha256_prefix":"3dee3c857bf38683","entry":"GeGLU","repo":"sizhelee/Diff-BGM","repo_kind":"official","path":"diffbgm/stable_diffusion/model/unet.py","file_url":"https://github.com/sizhelee/Diff-BGM/blob/HEAD/diffbgm/stable_diffusion/model/unet.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"3dee3c857bf38683"}},{"code_sha256_prefix":"31bf0a331a19cf4b","entry":"GroupNorm32","repo":"sizhelee/Diff-BGM","repo_kind":"official","path":"diffbgm/stable_diffusion/model/unet.py","file_url":"https://github.com/sizhelee/Diff-BGM/blob/HEAD/diffbgm/stable_diffusion/model/unet.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"31bf0a331a19cf4b"}},{"code_sha256_prefix":"501aecff0cc44374","entry":"UpSample","repo":"sizhelee/Diff-BGM","repo_kind":"official","path":"diffbgm/stable_diffusion/model/unet.py","file_url":"https://github.com/sizhelee/Diff-BGM/blob/HEAD/diffbgm/stable_diffusion/model/unet.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"501aecff0cc44374"}},{"code_sha256_prefix":"35e508b45fd23d2e","entry":"dummy_cond_input","repo":"sizhelee/diff-bgm","repo_kind":"official","path":"diffbgm/inference_sdf.py","file_url":"https://github.com/sizhelee/diff-bgm/blob/HEAD/diffbgm/inference_sdf.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"35e508b45fd23d2e"}},{"code_sha256_prefix":"874ebc0bd098eb14","entry":"normalization","repo":"sizhelee/Diff-BGM","repo_kind":"official","path":"diffbgm/stable_diffusion/model/unet.py","file_url":"https://github.com/sizhelee/Diff-BGM/blob/HEAD/diffbgm/stable_diffusion/model/unet.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"874ebc0bd098eb14"}},{"code_sha256_prefix":"9e853e757414f297","entry":"BasicTransformerBlock","repo":"sizhelee/Diff-BGM","repo_kind":"official","path":"diffbgm/stable_diffusion/model/unet.py","file_url":"https://github.com/sizhelee/Diff-BGM/blob/HEAD/diffbgm/stable_diffusion/model/unet.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"9e853e757414f297"}},{"code_sha256_prefix":"dc86d4f1807d75aa","entry":"ResBlock","repo":"sizhelee/Diff-BGM","repo_kind":"official","path":"diffbgm/stable_diffusion/model/unet.py","file_url":"https://github.com/sizhelee/Diff-BGM/blob/HEAD/diffbgm/stable_diffusion/model/unet.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"dc86d4f1807d75aa"}},{"code_sha256_prefix":"807d93291a28f2e7","entry":"SpatialTransformer","repo":"sizhelee/Diff-BGM","repo_kind":"official","path":"diffbgm/stable_diffusion/model/unet.py","file_url":"https://github.com/sizhelee/Diff-BGM/blob/HEAD/diffbgm/stable_diffusion/model/unet.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"807d93291a28f2e7"}},{"code_sha256_prefix":"324fc9cc3ad487cd","entry":"TimestepEmbedSequential","repo":"sizhelee/Diff-BGM","repo_kind":"official","path":"diffbgm/stable_diffusion/model/unet.py","file_url":"https://github.com/sizhelee/Diff-BGM/blob/HEAD/diffbgm/stable_diffusion/model/unet.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"324fc9cc3ad487cd"}},{"code_sha256_prefix":"ae15fb19407462fb","entry":"UNetModel","repo":"sizhelee/Diff-BGM","repo_kind":"official","path":"diffbgm/stable_diffusion/model/unet.py","file_url":"https://github.com/sizhelee/Diff-BGM/blob/HEAD/diffbgm/stable_diffusion/model/unet.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"ae15fb19407462fb"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}