{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/videocomposer-compositional-video-synthesis","title":"VideoComposer: Compositional Video Synthesis with Motion Controllability","arxiv_id":"2306.02018","date":"2023-06-03","proceeding":"NeurIPS 2023 11","authors":["Xiang Wang","Hangjie Yuan","Shiwei Zhang","Dayou Chen","Jiuniu Wang","Yingya Zhang","Yujun Shen","Deli Zhao","Jingren Zhou"],"abstract":"The pursuit of controllability as a higher standard of visual content creation has yielded remarkable progress in customizable image synthesis. However, achieving controllable video synthesis remains challenging due to the large variation of temporal dynamics and the requirement of cross-frame temporal consistency. Based on the paradigm of compositional generation, this work presents VideoComposer that allows users to flexibly compose a video with textual conditions, spatial conditions, and more importantly temporal conditions. Specifically, considering the characteristic of video data, we introduce the motion vector from compressed videos as an explicit control signal to provide guidance regarding temporal dynamics. In addition, we develop a Spatio-Temporal Condition encoder (STC-encoder) that serves as a unified interface to effectively incorporate the spatial and temporal relations of sequential inputs, with which the model could make better use of temporal conditions and hence achieve higher inter-frame consistency. Extensive experimental results suggest that VideoComposer is able to control the spatial and temporal patterns simultaneously within a synthesized video in various forms, such as text description, sketch sequence, reference video, or even simply hand-crafted motions. The code and models will be publicly available at https://videocomposer.github.io.","url_abs":"https://arxiv.org/abs/2306.02018v2","url_pdf":"https://arxiv.org/pdf/2306.02018v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"videocomposer-compositional-video-synthesis","repo_url":"https://github.com/ali-vilab/videocomposer","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"videocomposer-compositional-video-synthesis","repo_url":"https://github.com/damo-vilab/videocomposer","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"videocomposer-compositional-video-synthesis","repo_url":"https://github.com/ali-vilab/i2vgen-xl","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok"}},{"paper_slug":"videocomposer-compositional-video-synthesis","repo_url":"https://github.com/mindspore-lab/mindone","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"mindspore","reach":{"status":"gone","observed_at":"2026-09-17","how":"tree_404+repo_404"}}],"tasks":[{"task_slug":"image-generation","task_name":"Image Generation"},{"task_slug":"text-to-video-generation","task_name":"Text-to-Video Generation"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/text-to-video-generation-on-evalcrafter-text","task":"Text-to-Video Generation","dataset":"EvalCrafter Text-to-Video (ECTV) Dataset","model":"ModelScope","rank_in_archive_order":5,"of":5,"metrics":{"Motion Quality":"53.09","Temporal Consistency":"54.46","Text-to-Video Alignment":"57.8","Total Score":"218","Visual Quality":"52.47"},"uses_additional_data":true},{"leaderboard":"/sota/text-to-video-generation-on-msr-vtt","task":"Text-to-Video Generation","dataset":"MSR-VTT","model":"VideoComposer","rank_in_archive_order":10,"of":18,"metrics":{"CLIPSIM":"0.2932","FVD":"580"},"uses_additional_data":false}],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2306.02018","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2306.02018"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/ali-vilab/videocomposer","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/ali-vilab/i2vgen-xl","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/mindspore-lab/mindone","reach":{"status":"gone","observed_at":"2026-09-17","how":"tree_404+repo_404"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/damo-vilab/videocomposer","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran_draft_wrong":3,"ran_fixture":1,"ran":1,"unverified":16},"by_repo_kind":{"listed":{"samples":20,"ran":4,"repositories":2}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":1,"samples":[{"code_sha256_prefix":"6a35b62fbc80f70a","entry":"interpolate_fn","repo":"damo-vilab/videocomposer","repo_kind":"listed","path":"artist/ops/dpm_solver.py","file_url":"https://github.com/damo-vilab/videocomposer/blob/HEAD/artist/ops/dpm_solver.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":2,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"6a35b62fbc80f70a"}},{"code_sha256_prefix":"9fcdaa6e423e8aa7","entry":"Normalize","repo":"ali-vilab/videocomposer","repo_kind":"listed","path":"tools/videocomposer/autoencoder.py","file_url":"https://github.com/ali-vilab/videocomposer/blob/HEAD/tools/videocomposer/autoencoder.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"9fcdaa6e423e8aa7"}},{"code_sha256_prefix":"0e68c8747f1cfcd9","entry":"model_wrapper","repo":"damo-vilab/videocomposer","repo_kind":"listed","path":"artist/ops/dpm_solver.py","file_url":"https://github.com/damo-vilab/videocomposer/blob/HEAD/artist/ops/dpm_solver.py","link_basis":"harvester_set","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"0e68c8747f1cfcd9"}},{"code_sha256_prefix":"3137073275f8c21a","entry":"nonlinearity","repo":null,"repo_kind":null,"path":null,"file_url":null,"link_basis":"identical_code_first_harvested_elsewhere","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":null,"inline_ok":false,"mcp_get_code":{"code_sha256":"3137073275f8c21a"}},{"code_sha256_prefix":"9aa26711dc747d66","entry":"uint2single","repo":"damo-vilab/videocomposer","repo_kind":"listed","path":"artist/ops/degration.py","file_url":"https://github.com/damo-vilab/videocomposer/blob/HEAD/artist/ops/degration.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"9aa26711dc747d66"}},{"code_sha256_prefix":"a1ba6b0c4d96716f","entry":"all_gather","repo":"damo-vilab/videocomposer","repo_kind":"listed","path":"utils/distributed.py","file_url":"https://github.com/damo-vilab/videocomposer/blob/HEAD/utils/distributed.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a1ba6b0c4d96716f"}},{"code_sha256_prefix":"439067335e7b67b5","entry":"all_reduce","repo":"damo-vilab/videocomposer","repo_kind":"listed","path":"utils/distributed.py","file_url":"https://github.com/damo-vilab/videocomposer/blob/HEAD/utils/distributed.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"439067335e7b67b5"}},{"code_sha256_prefix":"6310b674ca6c549f","entry":"beta_schedule","repo":"damo-vilab/videocomposer","repo_kind":"listed","path":"artist/ops/diffusion.py","file_url":"https://github.com/damo-vilab/videocomposer/blob/HEAD/artist/ops/diffusion.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"6310b674ca6c549f"}},{"code_sha256_prefix":"c0132499fc4c9979","entry":"discretized_gaussian_log_likelihood","repo":"damo-vilab/videocomposer","repo_kind":"listed","path":"artist/ops/losses.py","file_url":"https://github.com/damo-vilab/videocomposer/blob/HEAD/artist/ops/losses.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c0132499fc4c9979"}},{"code_sha256_prefix":"b6d6bcb982b768cc","entry":"get_logger","repo":"damo-vilab/videocomposer","repo_kind":"listed","path":"utils/logging.py","file_url":"https://github.com/damo-vilab/videocomposer/blob/HEAD/utils/logging.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b6d6bcb982b768cc"}},{"code_sha256_prefix":"1d504697143a06b9","entry":"get_rank","repo":"damo-vilab/videocomposer","repo_kind":"listed","path":"artist/ops/distributed.py","file_url":"https://github.com/damo-vilab/videocomposer/blob/HEAD/artist/ops/distributed.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"1d504697143a06b9"}},{"code_sha256_prefix":"151fc386a67a4f69","entry":"get_world_size","repo":"damo-vilab/videocomposer","repo_kind":"listed","path":"artist/ops/distributed.py","file_url":"https://github.com/damo-vilab/videocomposer/blob/HEAD/artist/ops/distributed.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"151fc386a67a4f69"}},{"code_sha256_prefix":"57ba5b9dc5dfa8d3","entry":"imread_uint","repo":"damo-vilab/videocomposer","repo_kind":"listed","path":"artist/ops/degration.py","file_url":"https://github.com/damo-vilab/videocomposer/blob/HEAD/artist/ops/degration.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"57ba5b9dc5dfa8d3"}},{"code_sha256_prefix":"30fda522caa99e6e","entry":"is_master_proc","repo":"damo-vilab/videocomposer","repo_kind":"listed","path":"utils/distributed.py","file_url":"https://github.com/damo-vilab/videocomposer/blob/HEAD/utils/distributed.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"30fda522caa99e6e"}},{"code_sha256_prefix":"24f8868cf6825433","entry":"kl_divergence","repo":"damo-vilab/videocomposer","repo_kind":"listed","path":"artist/ops/losses.py","file_url":"https://github.com/damo-vilab/videocomposer/blob/HEAD/artist/ops/losses.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"24f8868cf6825433"}},{"code_sha256_prefix":"75297920dd4b445d","entry":"midas_v3","repo":"damo-vilab/videocomposer","repo_kind":"listed","path":"artist/models/midas.py","file_url":"https://github.com/damo-vilab/videocomposer/blob/HEAD/artist/models/midas.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"75297920dd4b445d"}},{"code_sha256_prefix":"c877d9da577fc0d3","entry":"model_wrapper_guided_diffusion","repo":"damo-vilab/videocomposer","repo_kind":"listed","path":"artist/ops/dpm_solver.py","file_url":"https://github.com/damo-vilab/videocomposer/blob/HEAD/artist/ops/dpm_solver.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c877d9da577fc0d3"}},{"code_sha256_prefix":"69723155a0580292","entry":"new_group","repo":"damo-vilab/videocomposer","repo_kind":"listed","path":"artist/ops/distributed.py","file_url":"https://github.com/damo-vilab/videocomposer/blob/HEAD/artist/ops/distributed.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"69723155a0580292"}},{"code_sha256_prefix":"3194e97648f3e21e","entry":"random_resize","repo":"damo-vilab/videocomposer","repo_kind":"listed","path":"gen_sketch.py","file_url":"https://github.com/damo-vilab/videocomposer/blob/HEAD/gen_sketch.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"3194e97648f3e21e"}},{"code_sha256_prefix":"83caf98edcef2e94","entry":"single2uint","repo":"damo-vilab/videocomposer","repo_kind":"listed","path":"artist/ops/degration.py","file_url":"https://github.com/damo-vilab/videocomposer/blob/HEAD/artist/ops/degration.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"83caf98edcef2e94"}},{"code_sha256_prefix":"59f8d2f892f3195a","entry":"standard_normal_cdf","repo":"damo-vilab/videocomposer","repo_kind":"listed","path":"artist/ops/losses.py","file_url":"https://github.com/damo-vilab/videocomposer/blob/HEAD/artist/ops/losses.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"59f8d2f892f3195a"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}