{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/representation-alignment-for-generation","title":"Representation Alignment for Generation: Training Diffusion Transformers Is Easier Than You Think","arxiv_id":"2410.06940","date":"2024-10-09","proceeding":null,"authors":["Sihyun Yu","Sangkyung Kwak","Huiwon Jang","Jongheon Jeong","Jonathan Huang","Jinwoo Shin","Saining Xie"],"abstract":"Recent studies have shown that the denoising process in (generative) diffusion models can induce meaningful (discriminative) representations inside the model, though the quality of these representations still lags behind those learned through recent self-supervised learning methods. We argue that one main bottleneck in training large-scale diffusion models for generation lies in effectively learning these representations. Moreover, training can be made easier by incorporating high-quality external visual representations, rather than relying solely on the diffusion models to learn them independently. We study this by introducing a straightforward regularization called REPresentation Alignment (REPA), which aligns the projections of noisy input hidden states in denoising networks with clean image representations obtained from external, pretrained visual encoders. The results are striking: our simple strategy yields significant improvements in both training efficiency and generation quality when applied to popular diffusion and flow-based transformers, such as DiTs and SiTs. For instance, our method can speed up SiT training by over 17.5$\\times$, matching the performance (without classifier-free guidance) of a SiT-XL model trained for 7M steps in less than 400K steps. In terms of final generation quality, our approach achieves state-of-the-art results of FID=1.42 using classifier-free guidance with the guidance interval.","url_abs":"https://arxiv.org/abs/2410.06940v2","url_pdf":"https://arxiv.org/pdf/2410.06940v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"representation-alignment-for-generation","repo_url":"https://github.com/sihyun-yu/REPA","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"denoising","task_name":"Denoising"},{"task_slug":"image-generation","task_name":"Image Generation"},{"task_slug":"self-supervised-learning","task_name":"Self-Supervised Learning"}],"methods":[{"method_slug":"diffusion","method_name":"Diffusion"},{"method_slug":"speed","method_name":"SPEED"}],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/image-generation-on-imagenet-256x256","task":"Image Generation","dataset":"ImageNet 256x256","model":"SiT-XL/2 + REPA (with the guidance interval)","rank_in_archive_order":18,"of":94,"metrics":{"FID":"1.42"},"uses_additional_data":false}],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2410.06940","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2410.06940"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/sihyun-yu/REPA","reach":null}],"summary":{"ran_draft_wrong":7,"unverified":2},"by_repo_kind":{"official":{"samples":3,"ran":2,"repositories":1},"community":{"samples":6,"ran":5,"repositories":2}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":2,"samples":[{"code_sha256_prefix":"7b21a01ae77703a3","entry":"create_npz_from_sample_folder","repo":"sihyun-yu/REPA","repo_kind":"official","path":"generate.py","file_url":"https://github.com/sihyun-yu/REPA/blob/HEAD/generate.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"7b21a01ae77703a3"}},{"code_sha256_prefix":"dd96bc6c07c85dae","entry":"mean_flat","repo":"sihyun-yu/REPA","repo_kind":"official","path":"loss.py","file_url":"https://github.com/sihyun-yu/REPA/blob/HEAD/loss.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"dd96bc6c07c85dae"}},{"code_sha256_prefix":"2574faa65a90497c","entry":"register_forward_hook","repo":"cloneofsimo/repa-rf","repo_kind":"community","path":"model.py","file_url":"https://github.com/cloneofsimo/repa-rf/blob/HEAD/model.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"2574faa65a90497c"}},{"code_sha256_prefix":"633782fa73a481c8","entry":"repeat_interleave_batch","repo":"End2End-Diffusion/REPA-E","repo_kind":"community","path":"models/jepa.py","file_url":"https://github.com/End2End-Diffusion/REPA-E/blob/HEAD/models/jepa.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"633782fa73a481c8"}},{"code_sha256_prefix":"d9ed023467b54ff6","entry":"sample_posterior","repo":"End2End-Diffusion/REPA-E","repo_kind":"community","path":"train_repae.py","file_url":"https://github.com/End2End-Diffusion/REPA-E/blob/HEAD/train_repae.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d9ed023467b54ff6"}},{"code_sha256_prefix":"226b182ef184aeeb","entry":"timestep_embedding","repo":"cloneofsimo/repa-rf","repo_kind":"community","path":"model.py","file_url":"https://github.com/cloneofsimo/repa-rf/blob/HEAD/model.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"226b182ef184aeeb"}},{"code_sha256_prefix":"28455f5511c6676f","entry":"trunc_normal_","repo":"End2End-Diffusion/REPA-E","repo_kind":"community","path":"models/jepa.py","file_url":"https://github.com/End2End-Diffusion/REPA-E/blob/HEAD/models/jepa.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"28455f5511c6676f"}},{"code_sha256_prefix":"d329e84f446c15f7","entry":"SILoss","repo":"sihyun-yu/REPA","repo_kind":"official","path":"loss.py","file_url":"https://github.com/sihyun-yu/REPA/blob/HEAD/loss.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d329e84f446c15f7"}},{"code_sha256_prefix":"d93366188784217e","entry":"apply_masks","repo":"End2End-Diffusion/REPA-E","repo_kind":"community","path":"models/jepa.py","file_url":"https://github.com/End2End-Diffusion/REPA-E/blob/HEAD/models/jepa.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d93366188784217e"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}