{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/anomalous-diffusion-dynamics-of-learning-in","title":"Anomalous diffusion dynamics of learning in deep neural networks","arxiv_id":"2009.10588","date":"2020-09-22","proceeding":null,"authors":["Guozhang Chen","Cheng Kevin Qu","Pulin Gong"],"abstract":"Learning in deep neural networks (DNNs) is implemented through minimizing a highly non-convex loss function, typically by a stochastic gradient descent (SGD) method. This learning process can effectively find good wide minima without being trapped in poor local ones. We present a novel account of how such effective deep learning emerges through the interactions of the SGD and the geometrical structure of the loss landscape. Rather than being a normal diffusion process (i.e. Brownian motion) as often assumed, we find that the SGD exhibits rich, complex dynamics when navigating through the loss landscape; initially, the SGD exhibits anomalous superdiffusion, which attenuates gradually and changes to subdiffusion at long times when the solution is reached. Such learning dynamics happen ubiquitously in different DNNs such as ResNet and VGG-like networks and are insensitive to batch size and learning rate. The anomalous superdiffusion process during the initial learning phase indicates that the motion of SGD along the loss landscape possesses intermittent, big jumps; this non-equilibrium property enables the SGD to escape from sharp local minima. By adapting the methods developed for studying energy landscapes in complex physical systems, we find that such superdiffusive learning dynamics are due to the interactions of the SGD and the fractal-like structure of the loss landscape. We further develop a simple model to demonstrate the mechanistic role of the fractal loss landscape in enabling the SGD to effectively find global minima. Our results thus reveal the effectiveness of deep learning from a novel perspective and have implications for designing efficient deep neural networks.","url_abs":"https://arxiv.org/abs/2009.10588v2","url_pdf":"https://arxiv.org/pdf/2009.10588v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"anomalous-diffusion-dynamics-of-learning-in","repo_url":"https://github.com/ifgovh/Anomalous-diffusion-dynamics-of-SGD","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[],"methods":[{"method_slug":"1x1-convolution","method_name":"1x1 Convolution"},{"method_slug":"average-pooling","method_name":"Average Pooling"},{"method_slug":"batch-normalization","method_name":"Batch Normalization"},{"method_slug":"bottleneck-residual-block","method_name":"Bottleneck Residual Block"},{"method_slug":"convolution","method_name":"Convolution"},{"method_slug":"diffusion","method_name":"Diffusion"},{"method_slug":"global-average-pooling","method_name":"Global Average Pooling"},{"method_slug":"kaiming-initialization","method_name":"Kaiming Initialization"},{"method_slug":"max-pooling","method_name":"Max Pooling"},{"method_slug":"relu","method_name":"ReLU"},{"method_slug":"residual-block","method_name":"Residual Block"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"sgd","method_name":"SGD"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2009.10588","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2009.10588"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/ifgovh/Anomalous-diffusion-dynamics-of-SGD","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"unverified":11},"by_repo_kind":{"official":{"samples":11,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"a69670bf9807e661","entry":"alpha_estimator","repo":"ifgovh/Anomalous-diffusion-dynamics-of-SGD","repo_kind":"official","path":"get_gradient_weight/gradient_noise.py","file_url":"https://github.com/ifgovh/Anomalous-diffusion-dynamics-of-SGD/blob/HEAD/get_gradient_weight/gradient_noise.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a69670bf9807e661"}},{"code_sha256_prefix":"191831befedc50f5","entry":"get_data_loaders","repo":"ifgovh/Anomalous-diffusion-dynamics-of-SGD","repo_kind":"official","path":"train_DNN_code/dataloader.py","file_url":"https://github.com/ifgovh/Anomalous-diffusion-dynamics-of-SGD/blob/HEAD/train_DNN_code/dataloader.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"191831befedc50f5"}},{"code_sha256_prefix":"81aabb5979983838","entry":"get_grads","repo":"ifgovh/Anomalous-diffusion-dynamics-of-SGD","repo_kind":"official","path":"get_gradient_weight/gradient_noise.py","file_url":"https://github.com/ifgovh/Anomalous-diffusion-dynamics-of-SGD/blob/HEAD/get_gradient_weight/gradient_noise.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"81aabb5979983838"}},{"code_sha256_prefix":"1e49a19659aab31b","entry":"get_layerWise_norms","repo":"ifgovh/Anomalous-diffusion-dynamics-of-SGD","repo_kind":"official","path":"get_gradient_weight/gradient_noise.py","file_url":"https://github.com/ifgovh/Anomalous-diffusion-dynamics-of-SGD/blob/HEAD/get_gradient_weight/gradient_noise.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"1e49a19659aab31b"}},{"code_sha256_prefix":"7ead1a7c9a99873f","entry":"get_random_states","repo":"ifgovh/Anomalous-diffusion-dynamics-of-SGD","repo_kind":"official","path":"get_gradient_weight/net_plotter.py","file_url":"https://github.com/ifgovh/Anomalous-diffusion-dynamics-of-SGD/blob/HEAD/get_gradient_weight/net_plotter.py","link_basis":"plan_row","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"7ead1a7c9a99873f"}},{"code_sha256_prefix":"3e80953c5e1e0841","entry":"get_random_weights","repo":"ifgovh/Anomalous-diffusion-dynamics-of-SGD","repo_kind":"official","path":"get_gradient_weight/net_plotter.py","file_url":"https://github.com/ifgovh/Anomalous-diffusion-dynamics-of-SGD/blob/HEAD/get_gradient_weight/net_plotter.py","link_basis":"plan_row","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"3e80953c5e1e0841"}},{"code_sha256_prefix":"028de37884922bbd","entry":"get_relative_path","repo":"ifgovh/Anomalous-diffusion-dynamics-of-SGD","repo_kind":"official","path":"dataloader.py","file_url":"https://github.com/ifgovh/Anomalous-diffusion-dynamics-of-SGD/blob/HEAD/dataloader.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"028de37884922bbd"}},{"code_sha256_prefix":"3b068a8f9eb6416e","entry":"get_synthetic_gaussian_data_loaders","repo":"ifgovh/Anomalous-diffusion-dynamics-of-SGD","repo_kind":"official","path":"train_DNN_code/dataloader.py","file_url":"https://github.com/ifgovh/Anomalous-diffusion-dynamics-of-SGD/blob/HEAD/train_DNN_code/dataloader.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"3b068a8f9eb6416e"}},{"code_sha256_prefix":"5fa87577ccee9cca","entry":"get_weights","repo":"ifgovh/Anomalous-diffusion-dynamics-of-SGD","repo_kind":"official","path":"get_gradient_weight/net_plotter.py","file_url":"https://github.com/ifgovh/Anomalous-diffusion-dynamics-of-SGD/blob/HEAD/get_gradient_weight/net_plotter.py","link_basis":"plan_row","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"5fa87577ccee9cca"}},{"code_sha256_prefix":"c8c8dee0fe2c623e","entry":"load_dataset","repo":"ifgovh/Anomalous-diffusion-dynamics-of-SGD","repo_kind":"official","path":"dataloader.py","file_url":"https://github.com/ifgovh/Anomalous-diffusion-dynamics-of-SGD/blob/HEAD/dataloader.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c8c8dee0fe2c623e"}},{"code_sha256_prefix":"0d11ac9cea6805a3","entry":"read_list","repo":"ifgovh/Anomalous-diffusion-dynamics-of-SGD","repo_kind":"official","path":"get_gradient_weight/h5_util.py","file_url":"https://github.com/ifgovh/Anomalous-diffusion-dynamics-of-SGD/blob/HEAD/get_gradient_weight/h5_util.py","link_basis":"plan_row","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"0d11ac9cea6805a3"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}