{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/distillation-of-discrete-diffusion-through","title":"Distillation of Discrete Diffusion through Dimensional Correlations","arxiv_id":"2410.08709","date":"2024-10-11","proceeding":null,"authors":["Satoshi Hayakawa","Yuhta Takida","Masaaki Imaizumi","Hiromi Wakaki","Yuki Mitsufuji"],"abstract":"Diffusion models have demonstrated exceptional performances in various fields of generative modeling, but suffer from slow sampling speed due to their iterative nature. While this issue is being addressed in continuous domains, discrete diffusion models face unique challenges, particularly in capturing dependencies between elements (e.g., pixel relationships in image, sequential dependencies in language) mainly due to the computational cost of processing high-dimensional joint distributions. In this paper, (i) we propose \"mixture\" models for discrete diffusion that are capable of treating dimensional correlations while remaining scalable, and (ii) we provide a set of loss functions for distilling the iterations of existing models. Two primary theoretical insights underpin our approach: First, conventional models with element-wise independence can well approximate the data distribution, but essentially require {\\it many sampling steps}. Second, our loss functions enable the mixture models to distill such many-step conventional models into just a few steps by learning the dimensional correlations. Our experimental results show the effectiveness of the proposed method in distilling pretrained discrete diffusion models across image and language domains. The code used in the paper is available at https://github.com/sony/di4c .","url_abs":"https://arxiv.org/abs/2410.08709v4","url_pdf":"https://arxiv.org/pdf/2410.08709v4.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"distillation-of-discrete-diffusion-through","repo_url":"https://github.com/sony/di4c","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[],"methods":[{"method_slug":"diffusion","method_name":"Diffusion"},{"method_slug":"set","method_name":"SET"},{"method_slug":"speed","method_name":"SPEED"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2410.08709","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2410.08709"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/sony/di4c","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":3,"ran_draft_wrong":2,"unverified":2},"by_repo_kind":{"official":{"samples":7,"ran":5,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"9ab699e19259b9d3","entry":"custom_lamda","repo":"sony/di4c","repo_kind":"official","path":"maskgit-pytorch/Trainer/trainer.py","file_url":"https://github.com/sony/di4c/blob/HEAD/maskgit-pytorch/Trainer/trainer.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"9ab699e19259b9d3"}},{"code_sha256_prefix":"221b2d116fdf1032","entry":"get_obj_from_str","repo":"sony/di4c","repo_kind":"official","path":"maskgit-pytorch/Network/Taming/util.py","file_url":"https://github.com/sony/di4c/blob/HEAD/maskgit-pytorch/Network/Taming/util.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"221b2d116fdf1032"}},{"code_sha256_prefix":"07d90bd6ae1be7db","entry":"md5_hash","repo":"sony/di4c","repo_kind":"official","path":"maskgit-pytorch/Network/Taming/util.py","file_url":"https://github.com/sony/di4c/blob/HEAD/maskgit-pytorch/Network/Taming/util.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"07d90bd6ae1be7db"}},{"code_sha256_prefix":"78d02565c457a57b","entry":"remap_image_torch","repo":"sony/di4c","repo_kind":"official","path":"maskgit-pytorch/Metrics/sample_and_eval.py","file_url":"https://github.com/sony/di4c/blob/HEAD/maskgit-pytorch/Metrics/sample_and_eval.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"78d02565c457a57b"}},{"code_sha256_prefix":"d0434d43396aa8fb","entry":"transformer_timestep_embedding","repo":"sony/di4c","repo_kind":"official","path":"maskgit-pytorch/Network/transformer.py","file_url":"https://github.com/sony/di4c/blob/HEAD/maskgit-pytorch/Network/transformer.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d0434d43396aa8fb"}},{"code_sha256_prefix":"b0a49b09dfb49452","entry":"eval_loop","repo":"sony/di4c","repo_kind":"official","path":"sdtt/src/sdtt/harness_gpt2.py","file_url":"https://github.com/sony/di4c/blob/HEAD/sdtt/src/sdtt/harness_gpt2.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b0a49b09dfb49452"}},{"code_sha256_prefix":"ef52d142b2c30ac2","entry":"instantiate_from_config","repo":"sony/di4c","repo_kind":"official","path":"maskgit-pytorch/Network/Taming/util.py","file_url":"https://github.com/sony/di4c/blob/HEAD/maskgit-pytorch/Network/Taming/util.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ef52d142b2c30ac2"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}