{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/improved-techniques-for-training-consistency","title":"Improved Techniques for Training Consistency Models","arxiv_id":"2310.14189","date":"2023-10-22","proceeding":null,"authors":["Yang song","Prafulla Dhariwal"],"abstract":"Consistency models are a nascent family of generative models that can sample high quality data in one step without the need for adversarial training. Current consistency models achieve optimal sample quality by distilling from pre-trained diffusion models and employing learned metrics such as LPIPS. However, distillation limits the quality of consistency models to that of the pre-trained diffusion model, and LPIPS causes undesirable bias in evaluation. To tackle these challenges, we present improved techniques for consistency training, where consistency models learn directly from data without distillation. We delve into the theory behind consistency training and identify a previously overlooked flaw, which we address by eliminating Exponential Moving Average from the teacher consistency model. To replace learned metrics like LPIPS, we adopt Pseudo-Huber losses from robust statistics. Additionally, we introduce a lognormal noise schedule for the consistency training objective, and propose to double total discretization steps every set number of training iterations. Combined with better hyperparameter tuning, these modifications enable consistency models to achieve FID scores of 2.51 and 3.25 on CIFAR-10 and ImageNet $64\\times 64$ respectively in a single sampling step. These scores mark a 3.5$\\times$ and 4$\\times$ improvement compared to prior consistency training approaches. Through two-step sampling, we further reduce FID scores to 2.24 and 2.77 on these two datasets, surpassing those obtained via distillation in both one-step and two-step settings, while narrowing the gap between consistency models and other state-of-the-art generative models.","url_abs":"https://arxiv.org/abs/2310.14189v1","url_pdf":"https://arxiv.org/pdf/2310.14189v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"improved-techniques-for-training-consistency","repo_url":"https://github.com/Kinyugo/consistency_models","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"improved-techniques-for-training-consistency","repo_url":"https://github.com/LYMDLUT/improved_consistency_models_cifar10_pytorch","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"improved-techniques-for-training-consistency","repo_url":"https://github.com/Mosasaur5526/BCM-iCT-torch","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"image-generation","task_name":"Image Generation"}],"methods":[{"method_slug":"diffusion","method_name":"Diffusion"},{"method_slug":"set","method_name":"SET"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2310.14189","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2310.14189"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/Mosasaur5526/BCM-iCT-torch","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/LYMDLUT/improved_consistency_models_cifar10_pytorch","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/Kinyugo/consistency_models","reach":null}],"summary":{"ran_honours":4,"ran_draft_wrong":1},"by_repo_kind":{"listed":{"samples":5,"ran":5,"repositories":3}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":3,"samples":[{"code_sha256_prefix":"0ec29979d800502a","entry":"ema_decay_rate_schedule","repo":"LYMDLUT/improved_consistency_models_cifar10_pytorch","repo_kind":"listed","path":"consistency_models.py","file_url":"https://github.com/LYMDLUT/improved_consistency_models_cifar10_pytorch/blob/HEAD/consistency_models.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"0ec29979d800502a"}},{"code_sha256_prefix":"248e6d374975855e","entry":"improved_timesteps_schedule","repo":"LYMDLUT/improved_consistency_models_cifar10_pytorch","repo_kind":"listed","path":"consistency_models.py","file_url":"https://github.com/LYMDLUT/improved_consistency_models_cifar10_pytorch/blob/HEAD/consistency_models.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"248e6d374975855e"}},{"code_sha256_prefix":"7d20d97e01869d5e","entry":"improved_timesteps_schedule","repo":"Kinyugo/consistency_models","repo_kind":"listed","path":"consistency_models/consistency_models.py","file_url":"https://github.com/Kinyugo/consistency_models/blob/HEAD/consistency_models/consistency_models.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"7d20d97e01869d5e"}},{"code_sha256_prefix":"b0e4dda425036670","entry":"normalize","repo":"Mosasaur5526/BCM-iCT-torch","repo_kind":"listed","path":"iCT/cm/unet.py","file_url":"https://github.com/Mosasaur5526/BCM-iCT-torch/blob/HEAD/iCT/cm/unet.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b0e4dda425036670"}},{"code_sha256_prefix":"5d181060fa21e0cc","entry":"timesteps_schedule","repo":"LYMDLUT/improved_consistency_models_cifar10_pytorch","repo_kind":"listed","path":"consistency_models.py","file_url":"https://github.com/LYMDLUT/improved_consistency_models_cifar10_pytorch/blob/HEAD/consistency_models.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"5d181060fa21e0cc"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}