{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/motionaura-generating-high-quality-and-motion","title":"MotionAura: Generating High-Quality and Motion Consistent Videos using Discrete Diffusion","arxiv_id":"2410.07659","date":"2024-10-10","proceeding":null,"authors":["Onkar Susladkar","Jishu Sen Gupta","Chirag Sehgal","Sparsh Mittal","Rekha Singhal"],"abstract":"The spatio-temporal complexity of video data presents significant challenges in tasks such as compression, generation, and inpainting. We present four key contributions to address the challenges of spatiotemporal video processing. First, we introduce the 3D Mobile Inverted Vector-Quantization Variational Autoencoder (3D-MBQ-VAE), which combines Variational Autoencoders (VAEs) with masked token modeling to enhance spatiotemporal video compression. The model achieves superior temporal consistency and state-of-the-art (SOTA) reconstruction quality by employing a novel training strategy with full frame masking. Second, we present MotionAura, a text-to-video generation framework that utilizes vector-quantized diffusion models to discretize the latent space and capture complex motion dynamics, producing temporally coherent videos aligned with text prompts. Third, we propose a spectral transformer-based denoising network that processes video data in the frequency domain using the Fourier Transform. This method effectively captures global context and long-range dependencies for high-quality video generation and denoising. Lastly, we introduce a downstream task of Sketch Guided Video Inpainting. This task leverages Low-Rank Adaptation (LoRA) for parameter-efficient fine-tuning. Our models achieve SOTA performance on a range of benchmarks. Our work offers robust frameworks for spatiotemporal modeling and user-driven video content manipulation. We will release the code, datasets, and models in open-source.","url_abs":"https://arxiv.org/abs/2410.07659v1","url_pdf":"https://arxiv.org/pdf/2410.07659v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"motionaura-generating-high-quality-and-motion","repo_url":"https://github.com/CandleLabAI/MotionAura-ICLR-2025","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"jax","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"denoising","task_name":"Denoising"},{"task_slug":"quantization","task_name":"Quantization"},{"task_slug":"text-to-video-generation","task_name":"Text-to-Video Generation"},{"task_slug":"video-compression","task_name":"Video Compression"},{"task_slug":"video-generation","task_name":"Video Generation"},{"task_slug":"video-inpainting","task_name":"Video Inpainting"},{"task_slug":"parameter-efficient-fine-tuning","task_name":"parameter-efficient fine-tuning"}],"methods":[{"method_slug":"diffusion","method_name":"Diffusion"},{"method_slug":"pixel-prediction","method_name":"Inpainting"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2410.07659","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2410.07659"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/CandleLabAI/MotionAura-ICLR-2025","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran":2,"ran_fixture":1,"ran_draft_wrong":1},"by_repo_kind":{"official":{"samples":4,"ran":4,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"fde7881344a30b12","entry":"betas_for_alpha_bar","repo":"CandleLabAI/MotionAura-ICLR-2025","repo_kind":"official","path":"src/diffusers/schedulers/scheduling_ddim_motionaura.py","file_url":"https://github.com/CandleLabAI/MotionAura-ICLR-2025/blob/HEAD/src/diffusers/schedulers/scheduling_ddim_motionaura.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"fde7881344a30b12"}},{"code_sha256_prefix":"734fd43f6aaef115","entry":"get_resize_crop_region_for_grid","repo":"CandleLabAI/MotionAura-ICLR-2025","repo_kind":"official","path":"src/diffusers/pipelines/motionaura/pipeline_motionaura.py","file_url":"https://github.com/CandleLabAI/MotionAura-ICLR-2025/blob/HEAD/src/diffusers/pipelines/motionaura/pipeline_motionaura.py","link_basis":"harvester_set","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"well_formed","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"734fd43f6aaef115"}},{"code_sha256_prefix":"4856ff8ee75aece7","entry":"rescale_zero_terminal_snr","repo":"CandleLabAI/MotionAura-ICLR-2025","repo_kind":"official","path":"src/diffusers/schedulers/scheduling_ddim_motionaura.py","file_url":"https://github.com/CandleLabAI/MotionAura-ICLR-2025/blob/HEAD/src/diffusers/schedulers/scheduling_ddim_motionaura.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"4856ff8ee75aece7"}},{"code_sha256_prefix":"22b1f260da28f6a5","entry":"retrieve_timesteps","repo":"CandleLabAI/MotionAura-ICLR-2025","repo_kind":"official","path":"src/diffusers/pipelines/motionaura/pipeline_motionaura.py","file_url":"https://github.com/CandleLabAI/MotionAura-ICLR-2025/blob/HEAD/src/diffusers/pipelines/motionaura/pipeline_motionaura.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"22b1f260da28f6a5"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}