{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/pyhessian-neural-networks-through-the-lens-of","title":"PyHessian: Neural Networks Through the Lens of the Hessian","arxiv_id":"1912.07145","date":"2019-12-16","proceeding":null,"authors":["Zhewei Yao","Amir Gholami","Kurt Keutzer","Michael Mahoney"],"abstract":"We present PYHESSIAN, a new scalable framework that enables fast computation of Hessian (i.e., second-order derivative) information for deep neural networks. PYHESSIAN enables fast computations of the top Hessian eigenvalues, the Hessian trace, and the full Hessian eigenvalue/spectral density, and it supports distributed-memory execution on cloud/supercomputer systems and is available as open source. This general framework can be used to analyze neural network models, including the topology of the loss landscape (i.e., curvature information) to gain insight into the behavior of different models/optimizers. To illustrate this, we analyze the effect of residual connections and Batch Normalization layers on the trainability of neural networks. One recent claim, based on simpler first-order analysis, is that residual connections and Batch Normalization make the loss landscape smoother, thus making it easier for Stochastic Gradient Descent to converge to a good solution. Our extensive analysis shows new finer-scale insights, demonstrating that, while conventional wisdom is sometimes validated, in other cases it is simply incorrect. In particular, we find that Batch Normalization does not necessarily make the loss landscape smoother, especially for shallower networks.","url_abs":"https://arxiv.org/abs/1912.07145v3","url_pdf":"https://arxiv.org/pdf/1912.07145v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"pyhessian-neural-networks-through-the-lens-of","repo_url":"https://github.com/amirgholami/pyhessian","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"pyhessian-neural-networks-through-the-lens-of","repo_url":"https://github.com/cxtraa/ngd_with_slt","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}},{"paper_slug":"pyhessian-neural-networks-through-the-lens-of","repo_url":"https://github.com/rmojgani/LPINNs","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[],"methods":[{"method_slug":"batch-normalization","method_name":"Batch Normalization"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/1912.07145","atlas_url":"https://app.syntology.ai/?focus=1912.07145","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"1912.07145"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/cxtraa/ngd_with_slt","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/amirgholami/pyhessian","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/rmojgani/LPINNs","reach":{"status":"ok"}}],"summary":{"ran_draft_wrong":1,"ran":5,"unverified":2},"by_repo_kind":{"official":{"samples":8,"ran":6,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"fac5364e2f53c6db","entry":"conv3x3","repo":"amirgholami/pyhessian","repo_kind":"official","path":"models/resnet.py","file_url":"https://github.com/amirgholami/pyhessian/blob/HEAD/models/resnet.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"fac5364e2f53c6db"}},{"code_sha256_prefix":"fed978b343667599","entry":"gaussian","repo":"amirgholami/pyhessian","repo_kind":"official","path":"density_plot.py","file_url":"https://github.com/amirgholami/pyhessian/blob/HEAD/density_plot.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"fed978b343667599"}},{"code_sha256_prefix":"2530e16bed546c6a","entry":"group_add","repo":"amirgholami/pyhessian","repo_kind":"official","path":"pyhessian/utils.py","file_url":"https://github.com/amirgholami/pyhessian/blob/HEAD/pyhessian/utils.py","link_basis":"plan_row","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"2530e16bed546c6a"}},{"code_sha256_prefix":"c35e05b48a3f5b73","entry":"group_product","repo":"amirgholami/pyhessian","repo_kind":"official","path":"pyhessian/utils.py","file_url":"https://github.com/amirgholami/pyhessian/blob/HEAD/pyhessian/utils.py","link_basis":"plan_row","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c35e05b48a3f5b73"}},{"code_sha256_prefix":"0e4f4ed57a87f93d","entry":"normalization","repo":"amirgholami/pyhessian","repo_kind":"official","path":"pyhessian/utils.py","file_url":"https://github.com/amirgholami/pyhessian/blob/HEAD/pyhessian/utils.py","link_basis":"plan_row","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"0e4f4ed57a87f93d"}},{"code_sha256_prefix":"efb74181eced6df4","entry":"test","repo":"amirgholami/pyhessian","repo_kind":"official","path":"utils.py","file_url":"https://github.com/amirgholami/pyhessian/blob/HEAD/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"efb74181eced6df4"}},{"code_sha256_prefix":"1425daa95c5d199f","entry":"density_generate","repo":"amirgholami/pyhessian","repo_kind":"official","path":"density_plot.py","file_url":"https://github.com/amirgholami/pyhessian/blob/HEAD/density_plot.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"1425daa95c5d199f"}},{"code_sha256_prefix":"1f5d5616297ed7a9","entry":"getData","repo":"amirgholami/pyhessian","repo_kind":"official","path":"utils.py","file_url":"https://github.com/amirgholami/pyhessian/blob/HEAD/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"1f5d5616297ed7a9"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}