{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/measurements-of-three-level-hierarchical","title":"Measurements of Three-Level Hierarchical Structure in the Outliers in the Spectrum of Deepnet Hessians","arxiv_id":"1901.08244","date":"2019-01-24","proceeding":null,"authors":["Vardan Papyan"],"abstract":"We consider deep classifying neural networks. We expose a structure in the\nderivative of the logits with respect to the parameters of the model, which is\nused to explain the existence of outliers in the spectrum of the Hessian.\nPrevious works decomposed the Hessian into two components, attributing the\noutliers to one of them, the so-called Covariance of gradients. We show this\nterm is not a Covariance but a second moment matrix, i.e., it is influenced by\nmeans of gradients. These means possess an additive two-way structure that is\nthe source of the outliers in the spectrum. This structure can be used to\napproximate the principal subspace of the Hessian using certain \"averaging\"\noperations, avoiding the need for high-dimensional eigenanalysis. We\ncorroborate this claim across different datasets, architectures and sample\nsizes.","url_abs":"http://arxiv.org/abs/1901.08244v1","url_pdf":"http://arxiv.org/pdf/1901.08244v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"measurements-of-three-level-hierarchical","repo_url":"https://github.com/deep-lab/DeepnetHessian","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/1901.08244","atlas_url":"https://app.syntology.ai/?focus=1901.08244","mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}