{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/accelerating-diffusion-transformers-with","title":"Accelerating Diffusion Transformers with Token-wise Feature Caching","arxiv_id":"2410.05317","date":"2024-10-05","proceeding":null,"authors":["Chang Zou","Xuyang Liu","Ting Liu","Siteng Huang","Linfeng Zhang"],"abstract":"Diffusion transformers have shown significant effectiveness in both image and video synthesis at the expense of huge computation costs. To address this problem, feature caching methods have been introduced to accelerate diffusion transformers by caching the features in previous timesteps and reusing them in the following timesteps. However, previous caching methods ignore that different tokens exhibit different sensitivities to feature caching, and feature caching on some tokens may lead to 10$\\times$ more destruction to the overall generation quality compared with other tokens. In this paper, we introduce token-wise feature caching, allowing us to adaptively select the most suitable tokens for caching, and further enable us to apply different caching ratios to neural layers in different types and depths. Extensive experiments on PixArt-$\\alpha$, OpenSora, and DiT demonstrate our effectiveness in both image and video generation with no requirements for training. For instance, 2.36$\\times$ and 1.93$\\times$ acceleration are achieved on OpenSora and PixArt-$\\alpha$ with almost no drop in generation quality.","url_abs":"https://arxiv.org/abs/2410.05317v3","url_pdf":"https://arxiv.org/pdf/2410.05317v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"accelerating-diffusion-transformers-with","repo_url":"https://github.com/Shenyi-Z/ToCa","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"accelerating-diffusion-transformers-with","repo_url":"https://github.com/shenyi-z/duca","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"video-generation","task_name":"Video Generation"}],"methods":[{"method_slug":"diffusion","method_name":"Diffusion"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2410.05317","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2410.05317"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/Shenyi-Z/ToCa","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/shenyi-z/duca","reach":null}],"summary":{"ran_honours":2,"ran":3,"ran_fixture":4,"ran_violates":1,"unverified":3},"by_repo_kind":{"official":{"samples":7,"ran":5,"repositories":1},"listed":{"samples":6,"ran":5,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":13,"samples":[{"code_sha256_prefix":"4e293dc87bdfee20","entry":"attn_score","repo":"Shenyi-Z/ToCa","repo_kind":"official","path":"PixArt-alpha-ToCa/diffusion/model/cache_functions/cache_cutfresh.py","file_url":"https://github.com/Shenyi-Z/ToCa/blob/HEAD/PixArt-alpha-ToCa/diffusion/model/cache_functions/cache_cutfresh.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"GPL-3.0","inline_ok":false,"mcp_get_code":{"code_sha256":"4e293dc87bdfee20"}},{"code_sha256_prefix":"346c3a66a40712b8","entry":"attn_score","repo":"shenyi-z/duca","repo_kind":"listed","path":"DuCa-DiT/cache_functions/cache_cutfresh.py","file_url":"https://github.com/shenyi-z/duca/blob/HEAD/DuCa-DiT/cache_functions/cache_cutfresh.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"GPL-3.0","inline_ok":false,"mcp_get_code":{"code_sha256":"346c3a66a40712b8"}},{"code_sha256_prefix":"197ebe37931d403b","entry":"fresh_ratio_scheduler","repo":"Shenyi-Z/ToCa","repo_kind":"official","path":"PixArt-alpha-ToCa/diffusion/model/cache_functions/cache_cutfresh.py","file_url":"https://github.com/Shenyi-Z/ToCa/blob/HEAD/PixArt-alpha-ToCa/diffusion/model/cache_functions/cache_cutfresh.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"GPL-3.0","inline_ok":false,"mcp_get_code":{"code_sha256":"197ebe37931d403b"}},{"code_sha256_prefix":"7cae5e5028551bec","entry":"fresh_ratio_scheduler","repo":"shenyi-z/duca","repo_kind":"listed","path":"DuCa-DiT/cache_functions/cache_cutfresh.py","file_url":"https://github.com/shenyi-z/duca/blob/HEAD/DuCa-DiT/cache_functions/cache_cutfresh.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"GPL-3.0","inline_ok":false,"mcp_get_code":{"code_sha256":"7cae5e5028551bec"}},{"code_sha256_prefix":"ab83bffc3af507ee","entry":"kv_norm_score","repo":"shenyi-z/duca","repo_kind":"listed","path":"DuCa-DiT/cache_functions/cache_cutfresh.py","file_url":"https://github.com/shenyi-z/duca/blob/HEAD/DuCa-DiT/cache_functions/cache_cutfresh.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"GPL-3.0","inline_ok":false,"mcp_get_code":{"code_sha256":"ab83bffc3af507ee"}},{"code_sha256_prefix":"bd60c5aeb25295c0","entry":"local_selection_with_bonus","repo":"Shenyi-Z/ToCa","repo_kind":"official","path":"PixArt-alpha-ToCa/diffusion/model/cache_functions/cache_cutfresh.py","file_url":"https://github.com/Shenyi-Z/ToCa/blob/HEAD/PixArt-alpha-ToCa/diffusion/model/cache_functions/cache_cutfresh.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"GPL-3.0","inline_ok":false,"mcp_get_code":{"code_sha256":"bd60c5aeb25295c0"}},{"code_sha256_prefix":"baf40c4c9808e3cd","entry":"local_selection_with_bonus","repo":"shenyi-z/duca","repo_kind":"listed","path":"DuCa-DiT/cache_functions/cache_cutfresh.py","file_url":"https://github.com/shenyi-z/duca/blob/HEAD/DuCa-DiT/cache_functions/cache_cutfresh.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"GPL-3.0","inline_ok":false,"mcp_get_code":{"code_sha256":"baf40c4c9808e3cd"}},{"code_sha256_prefix":"a2f810a673cf074a","entry":"norm_score","repo":"Shenyi-Z/ToCa","repo_kind":"official","path":"PixArt-alpha-ToCa/diffusion/model/cache_functions/cache_cutfresh.py","file_url":"https://github.com/Shenyi-Z/ToCa/blob/HEAD/PixArt-alpha-ToCa/diffusion/model/cache_functions/cache_cutfresh.py","link_basis":"first_harvest_node","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"GPL-3.0","inline_ok":false,"mcp_get_code":{"code_sha256":"a2f810a673cf074a"}},{"code_sha256_prefix":"9d8e492d65747dec","entry":"score_evaluate","repo":"shenyi-z/duca","repo_kind":"listed","path":"DuCa-DiT/cache_functions/cache_cutfresh.py","file_url":"https://github.com/shenyi-z/duca/blob/HEAD/DuCa-DiT/cache_functions/cache_cutfresh.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"GPL-3.0","inline_ok":false,"mcp_get_code":{"code_sha256":"9d8e492d65747dec"}},{"code_sha256_prefix":"195ca5d843403029","entry":"similarity_score","repo":"Shenyi-Z/ToCa","repo_kind":"official","path":"PixArt-alpha-ToCa/diffusion/model/cache_functions/cache_cutfresh.py","file_url":"https://github.com/Shenyi-Z/ToCa/blob/HEAD/PixArt-alpha-ToCa/diffusion/model/cache_functions/cache_cutfresh.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"GPL-3.0","inline_ok":false,"mcp_get_code":{"code_sha256":"195ca5d843403029"}},{"code_sha256_prefix":"bac2e25d384c61d9","entry":"cache_cutfresh","repo":"Shenyi-Z/ToCa","repo_kind":"official","path":"PixArt-alpha-ToCa/diffusion/model/cache_functions/cache_cutfresh.py","file_url":"https://github.com/Shenyi-Z/ToCa/blob/HEAD/PixArt-alpha-ToCa/diffusion/model/cache_functions/cache_cutfresh.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"GPL-3.0","inline_ok":false,"mcp_get_code":{"code_sha256":"bac2e25d384c61d9"}},{"code_sha256_prefix":"329975da8dd67e51","entry":"cache_cutfresh","repo":"shenyi-z/duca","repo_kind":"listed","path":"DuCa-DiT/cache_functions/cache_cutfresh.py","file_url":"https://github.com/shenyi-z/duca/blob/HEAD/DuCa-DiT/cache_functions/cache_cutfresh.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"GPL-3.0","inline_ok":false,"mcp_get_code":{"code_sha256":"329975da8dd67e51"}},{"code_sha256_prefix":"135b93cfa6587819","entry":"score_evaluate","repo":"Shenyi-Z/ToCa","repo_kind":"official","path":"PixArt-alpha-ToCa/diffusion/model/cache_functions/cache_cutfresh.py","file_url":"https://github.com/Shenyi-Z/ToCa/blob/HEAD/PixArt-alpha-ToCa/diffusion/model/cache_functions/cache_cutfresh.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"GPL-3.0","inline_ok":false,"mcp_get_code":{"code_sha256":"135b93cfa6587819"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}