{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/scalelong-towards-more-stable-training-of-1","title":"ScaleLong: Towards More Stable Training of Diffusion Model via Scaling Network Long Skip Connection","arxiv_id":"2310.13545","date":"2023-10-20","proceeding":"NeurIPS 2023 11","authors":["Zhongzhan Huang","Pan Zhou","Shuicheng Yan","Liang Lin"],"abstract":"In diffusion models, UNet is the most popular network backbone, since its long skip connects (LSCs) to connect distant network blocks can aggregate long-distant information and alleviate vanishing gradient. Unfortunately, UNet often suffers from unstable training in diffusion models which can be alleviated by scaling its LSC coefficients smaller. However, theoretical understandings of the instability of UNet in diffusion models and also the performance improvement of LSC scaling remain absent yet. To solve this issue, we theoretically show that the coefficients of LSCs in UNet have big effects on the stableness of the forward and backward propagation and robustness of UNet. Specifically, the hidden feature and gradient of UNet at any layer can oscillate and their oscillation ranges are actually large which explains the instability of UNet training. Moreover, UNet is also provably sensitive to perturbed input, and predicts an output distant from the desired output, yielding oscillatory loss and thus oscillatory gradient. Besides, we also observe the theoretical benefits of the LSC coefficient scaling of UNet in the stableness of hidden features and gradient and also robustness. Finally, inspired by our theory, we propose an effective coefficient scaling framework ScaleLong that scales the coefficients of LSC in UNet and better improves the training stability of UNet. Experimental results on four famous datasets show that our methods are superior to stabilize training and yield about 1.5x training acceleration on different diffusion models with UNet or UViT backbones. Code: https://github.com/sail-sg/ScaleLong","url_abs":"https://arxiv.org/abs/2310.13545v1","url_pdf":"https://arxiv.org/pdf/2310.13545v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"scalelong-towards-more-stable-training-of-1","repo_url":"https://github.com/sail-sg/scalelong","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":null},{"paper_slug":"scalelong-towards-more-stable-training-of-1","repo_url":"https://github.com/YichengDWu/tinyedm","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[],"methods":[{"method_slug":"diffusion","method_name":"Diffusion"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2310.13545","atlas_url":"https://app.syntology.ai/?focus=2310.13545","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2310.13545"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/sail-sg/scalelong","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/YichengDWu/tinyedm","reach":null}],"summary":{"ran":4,"ran_honours":1,"ran_draft_wrong":4,"ran_fixture":4},"by_repo_kind":{"official":{"samples":2,"ran":2,"repositories":1},"listed":{"samples":5,"ran":5,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":8,"samples":[{"code_sha256_prefix":"a21d05625a5f984b","entry":"Attention","repo":"sail-sg/scalelong","repo_kind":"official","path":"libs/uvit.py","file_url":"https://github.com/sail-sg/scalelong/blob/HEAD/libs/uvit.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"a21d05625a5f984b"}},{"code_sha256_prefix":"b4ecd978b7e2b6d9","entry":"Block","repo":"sail-sg/scalelong","repo_kind":"official","path":"libs/uvit.py","file_url":"https://github.com/sail-sg/scalelong/blob/HEAD/libs/uvit.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"b4ecd978b7e2b6d9"}},{"code_sha256_prefix":"2c4d769802a65492","entry":"Conv2d","repo":"YichengDWu/tinyedm","repo_kind":"listed","path":"src/tinyedm/networks.py","file_url":"https://github.com/YichengDWu/tinyedm/blob/HEAD/src/tinyedm/networks.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"2c4d769802a65492"}},{"code_sha256_prefix":"6c9210c75f7702f5","entry":"ScaleLong","repo":"YichengDWu/tinyedm","repo_kind":"listed","path":"src/tinyedm/networks.py","file_url":"https://github.com/YichengDWu/tinyedm/blob/HEAD/src/tinyedm/networks.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"6c9210c75f7702f5"}},{"code_sha256_prefix":"10b4cd4bf2697ea3","entry":"mp_silu","repo":"YichengDWu/tinyedm","repo_kind":"listed","path":"src/tinyedm/networks.py","file_url":"https://github.com/YichengDWu/tinyedm/blob/HEAD/src/tinyedm/networks.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"10b4cd4bf2697ea3"}},{"code_sha256_prefix":"2add1f3e88daf919","entry":"normalize","repo":"YichengDWu/tinyedm","repo_kind":"listed","path":"src/tinyedm/networks.py","file_url":"https://github.com/YichengDWu/tinyedm/blob/HEAD/src/tinyedm/networks.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"2add1f3e88daf919"}},{"code_sha256_prefix":"7a7de2a8e080f047","entry":"patchify","repo":null,"repo_kind":null,"path":null,"file_url":null,"link_basis":"identical_code_first_harvested_elsewhere","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":null,"inline_ok":false,"mcp_get_code":{"code_sha256":"7a7de2a8e080f047"}},{"code_sha256_prefix":"9cf076a2ae913f94","entry":"patchify","repo":null,"repo_kind":null,"path":null,"file_url":null,"link_basis":"identical_code_first_harvested_elsewhere","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":null,"inline_ok":false,"mcp_get_code":{"code_sha256":"9cf076a2ae913f94"}},{"code_sha256_prefix":"029abee41f338892","entry":"pixel_norm","repo":"YichengDWu/tinyedm","repo_kind":"listed","path":"src/tinyedm/networks.py","file_url":"https://github.com/YichengDWu/tinyedm/blob/HEAD/src/tinyedm/networks.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"029abee41f338892"}},{"code_sha256_prefix":"262b18d8278dcf70","entry":"timestep_embedding","repo":null,"repo_kind":null,"path":null,"file_url":null,"link_basis":"identical_code_first_harvested_elsewhere","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":null,"inline_ok":false,"mcp_get_code":{"code_sha256":"262b18d8278dcf70"}},{"code_sha256_prefix":"b494dd297ded5960","entry":"timestep_embedding","repo":null,"repo_kind":null,"path":null,"file_url":null,"link_basis":"identical_code_first_harvested_elsewhere","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":null,"inline_ok":false,"mcp_get_code":{"code_sha256":"b494dd297ded5960"}},{"code_sha256_prefix":"82e1ad8a9b00554e","entry":"unpatchify","repo":null,"repo_kind":null,"path":null,"file_url":null,"link_basis":"identical_code_first_harvested_elsewhere","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":null,"inline_ok":false,"mcp_get_code":{"code_sha256":"82e1ad8a9b00554e"}},{"code_sha256_prefix":"f9244c08c1011341","entry":"unpatchify","repo":null,"repo_kind":null,"path":null,"file_url":null,"link_basis":"identical_code_first_harvested_elsewhere","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":null,"inline_ok":false,"mcp_get_code":{"code_sha256":"f9244c08c1011341"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}