{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/tora-trajectory-oriented-diffusion","title":"Tora: Trajectory-oriented Diffusion Transformer for Video Generation","arxiv_id":"2407.21705","date":"2024-07-31","proceeding":"CVPR 2025 1","authors":["Zhenghao Zhang","Junchao Liao","Menghao Li","Zuozhuo Dai","Bingxue Qiu","Siyu Zhu","Long Qin","Weizhi Wang"],"abstract":"Recent advancements in Diffusion Transformer (DiT) have demonstrated remarkable proficiency in producing high-quality video content. Nonetheless, the potential of transformer-based diffusion models for effectively generating videos with controllable motion remains an area of limited exploration. This paper introduces Tora, the first trajectory-oriented DiT framework that concurrently integrates textual, visual, and trajectory conditions, thereby enabling scalable video generation with effective motion guidance. Specifically, Tora consists of a Trajectory Extractor(TE), a Spatial-Temporal DiT, and a Motion-guidance Fuser(MGF). The TE encodes arbitrary trajectories into hierarchical spacetime motion patches with a 3D video compression network. The MGF integrates the motion patches into the DiT blocks to generate consistent videos that accurately follow designated trajectories. Our design aligns seamlessly with DiT's scalability, allowing precise control of video content's dynamics with diverse durations, aspect ratios, and resolutions. Extensive experiments demonstrate Tora's excellence in achieving high motion fidelity, while also meticulously simulating the intricate movement of the physical world. Code is available at: https://github.com/alibaba/Tora.","url_abs":"https://arxiv.org/abs/2407.21705v3","url_pdf":"https://arxiv.org/pdf/2407.21705v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"tora-trajectory-oriented-diffusion","repo_url":"https://github.com/alibaba/Tora","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"video-compression","task_name":"Video Compression"},{"task_slug":"video-generation","task_name":"Video Generation"}],"methods":[{"method_slug":"absolute-position-encodings","method_name":"Absolute Position Encodings"},{"method_slug":"adam","method_name":"Adam"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"bpe","method_name":"BPE"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"diffusion","method_name":"Diffusion"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"label-smoothing","method_name":"Label Smoothing"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"position-wise-feed-forward-layer","method_name":"Position-Wise Feed-Forward Layer"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"transformer","method_name":"Transformer"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2407.21705","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2407.21705"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/alibaba/Tora","reach":null}],"summary":{"ran":6,"ran_violates":1,"ran_fixture":1,"ran_draft_wrong":1,"unverified":3},"by_repo_kind":{"official":{"samples":12,"ran":9,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"6a1cf7ce76caca59","entry":"Downsample","repo":"alibaba/Tora","repo_kind":"official","path":"diffusers-version/tora/transformer_3d.py","file_url":"https://github.com/alibaba/Tora/blob/HEAD/diffusers-version/tora/transformer_3d.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"6a1cf7ce76caca59"}},{"code_sha256_prefix":"bd3af06052e1a872","entry":"FloatGroupNorm","repo":"alibaba/Tora","repo_kind":"official","path":"diffusers-version/tora/transformer_3d.py","file_url":"https://github.com/alibaba/Tora/blob/HEAD/diffusers-version/tora/transformer_3d.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"bd3af06052e1a872"}},{"code_sha256_prefix":"c6b9d6647323596a","entry":"MGF","repo":"alibaba/Tora","repo_kind":"official","path":"diffusers-version/tora/transformer_3d.py","file_url":"https://github.com/alibaba/Tora/blob/HEAD/diffusers-version/tora/transformer_3d.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"c6b9d6647323596a"}},{"code_sha256_prefix":"62ae80ac1c524e6d","entry":"Patchify3D","repo":"alibaba/Tora","repo_kind":"official","path":"diffusers-version/tora/transformer_3d.py","file_url":"https://github.com/alibaba/Tora/blob/HEAD/diffusers-version/tora/transformer_3d.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"62ae80ac1c524e6d"}},{"code_sha256_prefix":"9dddf9902d99b405","entry":"ResnetBlock","repo":"alibaba/Tora","repo_kind":"official","path":"diffusers-version/tora/transformer_3d.py","file_url":"https://github.com/alibaba/Tora/blob/HEAD/diffusers-version/tora/transformer_3d.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"9dddf9902d99b405"}},{"code_sha256_prefix":"7e6c1654297bad79","entry":"TrajExtractor","repo":"alibaba/Tora","repo_kind":"official","path":"diffusers-version/tora/transformer_3d.py","file_url":"https://github.com/alibaba/Tora/blob/HEAD/diffusers-version/tora/transformer_3d.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"7e6c1654297bad79"}},{"code_sha256_prefix":"6eb26eb0c6d9315b","entry":"ensure_tuple","repo":"alibaba/Tora","repo_kind":"official","path":"diffusers-version/tora/transformer_3d.py","file_url":"https://github.com/alibaba/Tora/blob/HEAD/diffusers-version/tora/transformer_3d.py","link_basis":"first_harvest_node","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"6eb26eb0c6d9315b"}},{"code_sha256_prefix":"734fd43f6aaef115","entry":"get_resize_crop_region_for_grid","repo":"alibaba/Tora","repo_kind":"official","path":"diffusers-version/tora/i2v_pipeline.py","file_url":"https://github.com/alibaba/Tora/blob/HEAD/diffusers-version/tora/i2v_pipeline.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"well_formed","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"734fd43f6aaef115"}},{"code_sha256_prefix":"22b1f260da28f6a5","entry":"retrieve_timesteps","repo":"alibaba/Tora","repo_kind":"official","path":"diffusers-version/tora/i2v_pipeline.py","file_url":"https://github.com/alibaba/Tora/blob/HEAD/diffusers-version/tora/i2v_pipeline.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"22b1f260da28f6a5"}},{"code_sha256_prefix":"696621f779b45cc6","entry":"CogVideoXBlock","repo":"alibaba/Tora","repo_kind":"official","path":"diffusers-version/tora/transformer_3d.py","file_url":"https://github.com/alibaba/Tora/blob/HEAD/diffusers-version/tora/transformer_3d.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"696621f779b45cc6"}},{"code_sha256_prefix":"0d292a2ea249dbd7","entry":"ToraTransformer3DModel","repo":"alibaba/Tora","repo_kind":"official","path":"diffusers-version/tora/transformer_3d.py","file_url":"https://github.com/alibaba/Tora/blob/HEAD/diffusers-version/tora/transformer_3d.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"0d292a2ea249dbd7"}},{"code_sha256_prefix":"8262566e1ed7239a","entry":"retrieve_latents","repo":"alibaba/Tora","repo_kind":"official","path":"diffusers-version/tora/i2v_pipeline.py","file_url":"https://github.com/alibaba/Tora/blob/HEAD/diffusers-version/tora/i2v_pipeline.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"8262566e1ed7239a"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}