{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/stretching-each-dollar-diffusion-training","title":"Stretching Each Dollar: Diffusion Training from Scratch on a Micro-Budget","arxiv_id":"2407.15811","date":"2024-07-22","proceeding":"CVPR 2025 1","authors":["Vikash Sehwag","Xianghao Kong","Jingtao Li","Michael Spranger","Lingjuan Lyu"],"abstract":"As scaling laws in generative AI push performance, they also simultaneously concentrate the development of these models among actors with large computational resources. With a focus on text-to-image (T2I) generative models, we aim to address this bottleneck by demonstrating very low-cost training of large-scale T2I diffusion transformer models. As the computational cost of transformers increases with the number of patches in each image, we propose to randomly mask up to 75% of the image patches during training. We propose a deferred masking strategy that preprocesses all patches using a patch-mixer before masking, thus significantly reducing the performance degradation with masking, making it superior to model downscaling in reducing computational cost. We also incorporate the latest improvements in transformer architecture, such as the use of mixture-of-experts layers, to improve performance and further identify the critical benefit of using synthetic images in micro-budget training. Finally, using only 37M publicly available real and synthetic images, we train a 1.16 billion parameter sparse transformer with only \\$1,890 economical cost and achieve a 12.7 FID in zero-shot generation on the COCO dataset. Notably, our model achieves competitive FID and high-quality generations while incurring 118$\\times$ lower cost than stable diffusion models and 14$\\times$ lower cost than the current state-of-the-art approach that costs \\$28,400. We aim to release our end-to-end training pipeline to further democratize the training of large-scale diffusion models on micro-budgets.","url_abs":"https://arxiv.org/abs/2407.15811v1","url_pdf":"https://arxiv.org/pdf/2407.15811v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"stretching-each-dollar-diffusion-training","repo_url":"https://github.com/sonyresearch/micro_diffusion","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"mixture-of-experts","task_name":"Mixture-of-Experts"}],"methods":[{"method_slug":"adam","method_name":"Adam"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"attention-dropout","method_name":"Attention Dropout"},{"method_slug":"cosine-annealing","method_name":"Cosine Annealing"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"diffusion","method_name":"Diffusion"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"focus","method_name":"Focus"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"linear-warmup-with-cosine-annealing","method_name":"Linear Warmup With Cosine Annealing"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"sparse-transformer","method_name":"Sparse Transformer"},{"method_slug":"weight-decay","method_name":"Weight Decay"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2407.15811","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2407.15811"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/sonyresearch/micro_diffusion","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran":4,"unverified":2},"by_repo_kind":{"official":{"samples":6,"ran":4,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"0422be179c63df4c","entry":"collate_fn","repo":"sonyresearch/micro_diffusion","repo_kind":"official","path":"micro_diffusion/datasets/prepare/coco/convert.py","file_url":"https://github.com/sonyresearch/micro_diffusion/blob/HEAD/micro_diffusion/datasets/prepare/coco/convert.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"0422be179c63df4c"}},{"code_sha256_prefix":"7bf412e14072906e","entry":"create_norm","repo":"sonyresearch/micro_diffusion","repo_kind":"official","path":"micro_diffusion/models/utils.py","file_url":"https://github.com/sonyresearch/micro_diffusion/blob/HEAD/micro_diffusion/models/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"7bf412e14072906e"}},{"code_sha256_prefix":"383473c02ec387c5","entry":"modulate","repo":"sonyresearch/micro_diffusion","repo_kind":"official","path":"micro_diffusion/models/utils.py","file_url":"https://github.com/sonyresearch/micro_diffusion/blob/HEAD/micro_diffusion/models/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"383473c02ec387c5"}},{"code_sha256_prefix":"db15776e6b4101b4","entry":"ntuple","repo":"sonyresearch/micro_diffusion","repo_kind":"official","path":"micro_diffusion/models/utils.py","file_url":"https://github.com/sonyresearch/micro_diffusion/blob/HEAD/micro_diffusion/models/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"db15776e6b4101b4"}},{"code_sha256_prefix":"47ea919e7f532ed6","entry":"MicroDiT_Tiny_2","repo":"sonyresearch/micro_diffusion","repo_kind":"official","path":"micro_diffusion/models/dit.py","file_url":"https://github.com/sonyresearch/micro_diffusion/blob/HEAD/micro_diffusion/models/dit.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"47ea919e7f532ed6"}},{"code_sha256_prefix":"f67652acb143f58a","entry":"MicroDiT_XL_2","repo":"sonyresearch/micro_diffusion","repo_kind":"official","path":"micro_diffusion/models/dit.py","file_url":"https://github.com/sonyresearch/micro_diffusion/blob/HEAD/micro_diffusion/models/dit.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"f67652acb143f58a"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}