{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/an-investigation-into-neural-net-optimization","title":"An Investigation into Neural Net Optimization via Hessian Eigenvalue Density","arxiv_id":"1901.10159","date":"2019-01-29","proceeding":null,"authors":["Behrooz Ghorbani","Shankar Krishnan","Ying Xiao"],"abstract":"To understand the dynamics of optimization in deep neural networks, we\ndevelop a tool to study the evolution of the entire Hessian spectrum throughout\nthe optimization process. Using this, we study a number of hypotheses\nconcerning smoothness, curvature, and sharpness in the deep learning\nliterature. We then thoroughly analyze a crucial structural feature of the\nspectra: in non-batch normalized networks, we observe the rapid appearance of\nlarge isolated eigenvalues in the spectrum, along with a surprising\nconcentration of the gradient in the corresponding eigenspaces. In batch\nnormalized networks, these two effects are almost absent. We characterize these\neffects, and explain how they affect optimization speed through both theory and\nexperiments. As part of this work, we adapt advanced tools from numerical\nlinear algebra that allow scalable and accurate estimation of the entire\nHessian spectrum of ImageNet-scale neural networks; this technique may be of\nindependent interest in other applications.","url_abs":"http://arxiv.org/abs/1901.10159v1","url_pdf":"http://arxiv.org/pdf/1901.10159v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"an-investigation-into-neural-net-optimization","repo_url":"https://github.com/google/spectral-density","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"jax","reach":null}],"tasks":[],"methods":[{"method_slug":"speed","method_name":"SPEED"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":null,"atlas_url":"https://app.syntology.ai/?focus=1901.10159","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"1901.10159"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/google/spectral-density","reach":null}],"summary":{"unverified":1},"by_repo_kind":{"listed":{"samples":1,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"e320a2d57303dcca","entry":"lanczos_alg","repo":"google/spectral-density","repo_kind":"listed","path":"jax/lanczos.py","file_url":"https://github.com/google/spectral-density/blob/HEAD/jax/lanczos.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"e320a2d57303dcca"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}