{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/arxiv-2510-16927","title":"Closing the Curvature Gap: Full Transformer Hessians","arxiv_id":"2510.16927","date":"2025-10-19","proceeding":"ICLR","authors":["Egor Petrov","Nikita Kiselev","Vladislav Meshkov","Andrey Grabovoy"],"abstract":"The optimization landscape of Transformer models remains poorly understood despite their widespread adoption. While recent studies have derived curvature properties for isolated self-attention mechanisms, a comprehensive theoretical characterization of the full Transformer block, accounting for the interactions between Layer Normalization, Feed-Forward Networks (FFNs), and residual connections, is missing. In this work, we close this gap by deriving the exact, closed-form Hessian for the complete Transformer block under arbitrary twice-differentiable loss functions. We utilize rigorous matrix calculus to handle the non-linearities of LayerNorm and row-wise activations, establishing explicit spectral norm bounds for the resulting Hessian blocks. Our analysis reveals how different architectural components contribute distinct curvature mechanisms, identifying the specific curvature contributions of particular sub-layers. Furthermore, empirical validation against automatic differentiation confirms the exactness of the derived formulas up to numerical precision and shows substantial computational speedups for the closed-form Jacobian evaluations.","url_abs":"https://arxiv.org/abs/2510.16927","url_pdf":"https://arxiv.org/pdf/2510.16927","source":{"archive":null,"snapshot":"2025-07-28","note":"not in the Papers with Code archive (frozen at the snapshot)","row_kind":"graph","title_abstract_authors_date":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)"},"code_links":[],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2510.16927","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2510.16927"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"mentioned_in_github":null,"is_official":null,"provenance":"deterministic:regex_extraction","mentioned_in_paper":null,"url":"https://github.com/modernTalker/transformer_hessian","reach":{"status":"ok"}}],"summary":{"unverified":5},"by_repo_kind":{"found_in_text":{"samples":5,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":5,"samples":[{"code_sha256_prefix":"6701ca0c5a6a1527","entry":"get_cifar100_dataloader","repo":"modernTalker/transformer_hessian","repo_kind":"found_in_text","path":"src/datasets/cifar_100.py","file_url":"https://github.com/modernTalker/transformer_hessian/blob/HEAD/src/datasets/cifar_100.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"6701ca0c5a6a1527"}},{"code_sha256_prefix":"7449e056cb2deb52","entry":"get_mnist_dataloader","repo":"modernTalker/transformer_hessian","repo_kind":"found_in_text","path":"src/datasets/mnist.py","file_url":"https://github.com/modernTalker/transformer_hessian/blob/HEAD/src/datasets/mnist.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"7449e056cb2deb52"}},{"code_sha256_prefix":"ea73e4b9a711319f","entry":"get_vit","repo":"modernTalker/transformer_hessian","repo_kind":"found_in_text","path":"src/models/vit_classification.py","file_url":"https://github.com/modernTalker/transformer_hessian/blob/HEAD/src/models/vit_classification.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"ea73e4b9a711319f"}},{"code_sha256_prefix":"256a40082be601a0","entry":"load_trainer_checkpoint","repo":"modernTalker/transformer_hessian","repo_kind":"found_in_text","path":"src/train/checkpoint.py","file_url":"https://github.com/modernTalker/transformer_hessian/blob/HEAD/src/train/checkpoint.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"256a40082be601a0"}},{"code_sha256_prefix":"b4f5befe6b014144","entry":"validate","repo":"modernTalker/transformer_hessian","repo_kind":"found_in_text","path":"src/train/vit_classification.py","file_url":"https://github.com/modernTalker/transformer_hessian/blob/HEAD/src/train/vit_classification.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"b4f5befe6b014144"}}]},"arxiv_metadata":{"licence":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)","fields":["title","abstract","authors","date"],"primary_category":"cs.LG","source":"arxiv_api"},"syntology_extracted_results":null}