{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/insvie-1m-effective-instruction-based-video","title":"InsViE-1M: Effective Instruction-based Video Editing with Elaborate Dataset Construction","arxiv_id":"2503.20287","date":"2025-03-26","proceeding":null,"authors":["Yuhui Wu","Liyi Chen","Ruibin Li","Shihao Wang","Chenxi Xie","Lei Zhang"],"abstract":"Instruction-based video editing allows effective and interactive editing of videos using only instructions without extra inputs such as masks or attributes. However, collecting high-quality training triplets (source video, edited video, instruction) is a challenging task. Existing datasets mostly consist of low-resolution, short duration, and limited amount of source videos with unsatisfactory editing quality, limiting the performance of trained editing models. In this work, we present a high-quality Instruction-based Video Editing dataset with 1M triplets, namely InsViE-1M. We first curate high-resolution and high-quality source videos and images, then design an effective editing-filtering pipeline to construct high-quality editing triplets for model training. For a source video, we generate multiple edited samples of its first frame with different intensities of classifier-free guidance, which are automatically filtered by GPT-4o with carefully crafted guidelines. The edited first frame is propagated to subsequent frames to produce the edited video, followed by another round of filtering for frame quality and motion evaluation. We also generate and filter a variety of video editing triplets from high-quality images. With the InsViE-1M dataset, we propose a multi-stage learning strategy to train our InsViE model, progressively enhancing its instruction following and editing ability. Extensive experiments demonstrate the advantages of our InsViE-1M dataset and the trained model over state-of-the-art works. Codes are available at InsViE.","url_abs":"https://arxiv.org/abs/2503.20287v1","url_pdf":"https://arxiv.org/pdf/2503.20287v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"insvie-1m-effective-instruction-based-video","repo_url":"https://github.com/langmanbusi/insvie","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"instruction-following","task_name":"Instruction Following"},{"task_slug":"video-editing","task_name":"Video Editing"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":null,"atlas_url":"https://app.syntology.ai/?focus=2503.20287","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2503.20287"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/langmanbusi/insvie","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran_draft_wrong":1,"unverified":17},"by_repo_kind":{"official":{"samples":18,"ran":1,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"da7df331dab819db","entry":"get_unique_embedder_keys_from_conditioner","repo":"langmanbusi/insvie","repo_kind":"official","path":"CogVideo/sat/sample_video.py","file_url":"https://github.com/langmanbusi/insvie/blob/HEAD/CogVideo/sat/sample_video.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"da7df331dab819db"}},{"code_sha256_prefix":"ed0fa4fe4bafc350","entry":"add_model_config_args","repo":"langmanbusi/insvie","repo_kind":"official","path":"CogVideo/sat/arguments.py","file_url":"https://github.com/langmanbusi/insvie/blob/HEAD/CogVideo/sat/arguments.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ed0fa4fe4bafc350"}},{"code_sha256_prefix":"73a95ca1e2cff647","entry":"add_sampling_config_args","repo":"langmanbusi/insvie","repo_kind":"official","path":"CogVideo/sat/arguments.py","file_url":"https://github.com/langmanbusi/insvie/blob/HEAD/CogVideo/sat/arguments.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"73a95ca1e2cff647"}},{"code_sha256_prefix":"02c48da3e29b7180","entry":"compute_prompt_embeddings","repo":"langmanbusi/insvie","repo_kind":"official","path":"CogVideo/finetune/train_cogvideox_image_to_video_lora.py","file_url":"https://github.com/langmanbusi/insvie/blob/HEAD/CogVideo/finetune/train_cogvideox_image_to_video_lora.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"02c48da3e29b7180"}},{"code_sha256_prefix":"3f2fc3183e384950","entry":"convert_prompt","repo":"langmanbusi/insvie","repo_kind":"official","path":"CogVideo/inference/convert_demo.py","file_url":"https://github.com/langmanbusi/insvie/blob/HEAD/CogVideo/inference/convert_demo.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"3f2fc3183e384950"}},{"code_sha256_prefix":"55cbe394bd642f3b","entry":"convert_prompt","repo":"langmanbusi/insvie","repo_kind":"official","path":"CogVideo/inference/gradio_web_demo.py","file_url":"https://github.com/langmanbusi/insvie/blob/HEAD/CogVideo/inference/gradio_web_demo.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"55cbe394bd642f3b"}},{"code_sha256_prefix":"e1f46e5593d36bb6","entry":"encode_prompt","repo":"langmanbusi/insvie","repo_kind":"official","path":"CogVideo/finetune/train_cogvideox_image_to_video_lora.py","file_url":"https://github.com/langmanbusi/insvie/blob/HEAD/CogVideo/finetune/train_cogvideox_image_to_video_lora.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e1f46e5593d36bb6"}},{"code_sha256_prefix":"cdd29f549144d05e","entry":"get_2d_sincos_pos_embed","repo":"langmanbusi/insvie","repo_kind":"official","path":"CogVideo/sat/dit_video_concat.py","file_url":"https://github.com/langmanbusi/insvie/blob/HEAD/CogVideo/sat/dit_video_concat.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"cdd29f549144d05e"}},{"code_sha256_prefix":"665d8a4e8f673a4c","entry":"get_2d_sincos_pos_embed_from_grid","repo":"langmanbusi/insvie","repo_kind":"official","path":"CogVideo/sat/dit_video_concat.py","file_url":"https://github.com/langmanbusi/insvie/blob/HEAD/CogVideo/sat/dit_video_concat.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"665d8a4e8f673a4c"}},{"code_sha256_prefix":"7759e9a501ea471a","entry":"get_3d_sincos_pos_embed","repo":"langmanbusi/insvie","repo_kind":"official","path":"CogVideo/sat/dit_video_concat.py","file_url":"https://github.com/langmanbusi/insvie/blob/HEAD/CogVideo/sat/dit_video_concat.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"7759e9a501ea471a"}},{"code_sha256_prefix":"4f7e73ff9945218a","entry":"get_batch","repo":"langmanbusi/insvie","repo_kind":"official","path":"CogVideo/sat/sample_video.py","file_url":"https://github.com/langmanbusi/insvie/blob/HEAD/CogVideo/sat/sample_video.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"4f7e73ff9945218a"}},{"code_sha256_prefix":"e8653493c0e3da07","entry":"image_to_url","repo":"langmanbusi/insvie","repo_kind":"official","path":"CogVideo/inference/convert_demo.py","file_url":"https://github.com/langmanbusi/insvie/blob/HEAD/CogVideo/inference/convert_demo.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e8653493c0e3da07"}},{"code_sha256_prefix":"44d67662b32e0a93","entry":"make_inference","repo":"langmanbusi/insvie","repo_kind":"official","path":"CogVideo/inference/gradio_composite_demo/rife_model.py","file_url":"https://github.com/langmanbusi/insvie/blob/HEAD/CogVideo/inference/gradio_composite_demo/rife_model.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"44d67662b32e0a93"}},{"code_sha256_prefix":"7668884d687f766a","entry":"pad_image","repo":"langmanbusi/insvie","repo_kind":"official","path":"CogVideo/inference/gradio_composite_demo/rife_model.py","file_url":"https://github.com/langmanbusi/insvie/blob/HEAD/CogVideo/inference/gradio_composite_demo/rife_model.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"7668884d687f766a"}},{"code_sha256_prefix":"f15588b5788ebab2","entry":"pad_last_frame","repo":"langmanbusi/insvie","repo_kind":"official","path":"CogVideo/sat/data_video.py","file_url":"https://github.com/langmanbusi/insvie/blob/HEAD/CogVideo/sat/data_video.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f15588b5788ebab2"}},{"code_sha256_prefix":"ea5ba6b6fc22be49","entry":"read_video","repo":"langmanbusi/insvie","repo_kind":"official","path":"CogVideo/sat/data_video.py","file_url":"https://github.com/langmanbusi/insvie/blob/HEAD/CogVideo/sat/data_video.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ea5ba6b6fc22be49"}},{"code_sha256_prefix":"e0dd6c8c26886b58","entry":"resize_for_rectangle_crop","repo":"langmanbusi/insvie","repo_kind":"official","path":"CogVideo/sat/data_video.py","file_url":"https://github.com/langmanbusi/insvie/blob/HEAD/CogVideo/sat/data_video.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e0dd6c8c26886b58"}},{"code_sha256_prefix":"f104145895788f8e","entry":"save_video","repo":"langmanbusi/insvie","repo_kind":"official","path":"CogVideo/inference/gradio_web_demo.py","file_url":"https://github.com/langmanbusi/insvie/blob/HEAD/CogVideo/inference/gradio_web_demo.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f104145895788f8e"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}