{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/fam-relative-flatness-aware-minimization","title":"FAM: Relative Flatness Aware Minimization","arxiv_id":"2307.02337","date":"2023-07-05","proceeding":null,"authors":["Linara Adilova","Amr Abourayya","Jianning Li","Amin Dada","Henning Petzka","Jan Egger","Jens Kleesiek","Michael Kamp"],"abstract":"Flatness of the loss curve around a model at hand has been shown to empirically correlate with its generalization ability. Optimizing for flatness has been proposed as early as 1994 by Hochreiter and Schmidthuber, and was followed by more recent successful sharpness-aware optimization techniques. Their widespread adoption in practice, though, is dubious because of the lack of theoretically grounded connection between flatness and generalization, in particular in light of the reparameterization curse - certain reparameterizations of a neural network change most flatness measures but do not change generalization. Recent theoretical work suggests that a particular relative flatness measure can be connected to generalization and solves the reparameterization curse. In this paper, we derive a regularizer based on this relative flatness that is easy to compute, fast, efficient, and works with arbitrary loss functions. It requires computing the Hessian only of a single layer of the network, which makes it applicable to large neural networks, and with it avoids an expensive mapping of the loss surface in the vicinity of the model. In an extensive empirical evaluation we show that this relative flatness aware minimization (FAM) improves generalization in a multitude of applications and models, both in finetuning and standard training. We make the code available at github.","url_abs":"https://arxiv.org/abs/2307.02337v1","url_pdf":"https://arxiv.org/pdf/2307.02337v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"fam-relative-flatness-aware-minimization","repo_url":"https://github.com/kampmichael/RelativeFlatnessAndGeneralization","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[],"methods":[{"method_slug":"aware","method_name":"AWARE"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2307.02337","atlas_url":"https://app.syntology.ai/?focus=2307.02337","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2307.02337"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/kampmichael/RelativeFlatnessAndGeneralization","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran":4,"unverified":5},"by_repo_kind":{"official":{"samples":9,"ran":4,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"cd59b0fd8051ec17","entry":"FAMreg","repo":"kampmichael/RelativeFlatnessAndGeneralization","repo_kind":"official","path":"RelativeFlatnessRegularizer(FAM)/FAMloss.py","file_url":"https://github.com/kampmichael/RelativeFlatnessAndGeneralization/blob/HEAD/RelativeFlatnessRegularizer%28FAM%29/FAMloss.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"cd59b0fd8051ec17"}},{"code_sha256_prefix":"8466f1e2cecac5f1","entry":"calculate_loss_on_data","repo":"kampmichael/RelativeFlatnessAndGeneralization","repo_kind":"official","path":"RelativeFlatnessRegularizer(FAM)/FAMloss.py","file_url":"https://github.com/kampmichael/RelativeFlatnessAndGeneralization/blob/HEAD/RelativeFlatnessRegularizer%28FAM%29/FAMloss.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"8466f1e2cecac5f1"}},{"code_sha256_prefix":"535c69c58c3d899b","entry":"load_cifar10","repo":"kampmichael/RelativeFlatnessAndGeneralization","repo_kind":"official","path":"CorrelationFlatnessGeneralization/data_loaders.py","file_url":"https://github.com/kampmichael/RelativeFlatnessAndGeneralization/blob/HEAD/CorrelationFlatnessGeneralization/data_loaders.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"535c69c58c3d899b"}},{"code_sha256_prefix":"10cb584d3d906e2f","entry":"softmax_accuracy","repo":"kampmichael/RelativeFlatnessAndGeneralization","repo_kind":"official","path":"RelativeFlatnessRegularizer(FAM)/FAMloss.py","file_url":"https://github.com/kampmichael/RelativeFlatnessAndGeneralization/blob/HEAD/RelativeFlatnessRegularizer%28FAM%29/FAMloss.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"10cb584d3d906e2f"}},{"code_sha256_prefix":"d83dceb54f2e4cdd","entry":"assemble_compare_measures","repo":"kampmichael/RelativeFlatnessAndGeneralization","repo_kind":"official","path":"CorrelationFlatnessGeneralization/plot_assemble_results.py","file_url":"https://github.com/kampmichael/RelativeFlatnessAndGeneralization/blob/HEAD/CorrelationFlatnessGeneralization/plot_assemble_results.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"d83dceb54f2e4cdd"}},{"code_sha256_prefix":"7c6793c5676f54f5","entry":"calculateNeuronwiseHessians_fc_layer","repo":"kampmichael/RelativeFlatnessAndGeneralization","repo_kind":"official","path":"CorrelationFlatnessGeneralization/utils.py","file_url":"https://github.com/kampmichael/RelativeFlatnessAndGeneralization/blob/HEAD/CorrelationFlatnessGeneralization/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"7c6793c5676f54f5"}},{"code_sha256_prefix":"41675c5109a70a82","entry":"combineWeights","repo":"kampmichael/RelativeFlatnessAndGeneralization","repo_kind":"official","path":"SyntheticExperiments/reverseEngineerNetwork.py","file_url":"https://github.com/kampmichael/RelativeFlatnessAndGeneralization/blob/HEAD/SyntheticExperiments/reverseEngineerNetwork.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"41675c5109a70a82"}},{"code_sha256_prefix":"1b510324243cfd6c","entry":"getReverseEngineeredModel_ridge","repo":"kampmichael/RelativeFlatnessAndGeneralization","repo_kind":"official","path":"SyntheticExperiments/reverseEngineerNetwork.py","file_url":"https://github.com/kampmichael/RelativeFlatnessAndGeneralization/blob/HEAD/SyntheticExperiments/reverseEngineerNetwork.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"1b510324243cfd6c"}},{"code_sha256_prefix":"5b5a8f955effd35a","entry":"reverseModelPropagation","repo":"kampmichael/RelativeFlatnessAndGeneralization","repo_kind":"official","path":"SyntheticExperiments/reverseEngineerNetwork.py","file_url":"https://github.com/kampmichael/RelativeFlatnessAndGeneralization/blob/HEAD/SyntheticExperiments/reverseEngineerNetwork.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"5b5a8f955effd35a"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}