{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/in-search-of-probeable-generalization","title":"In Search of Probeable Generalization Measures","arxiv_id":"2110.12259","date":"2021-10-23","proceeding":null,"authors":["Jonathan Jaegerman","Khalil Damouni","Mahdi S. Hosseini","Konstantinos N. Plataniotis"],"abstract":"Understanding the generalization behaviour of deep neural networks is a topic of recent interest that has driven the production of many studies, notably the development and evaluation of generalization \"explainability\" measures that quantify model generalization ability. Generalization measures have also proven useful in the development of powerful layer-wise model tuning and optimization algorithms, though these algorithms require specific kinds of generalization measures which can probe individual layers. The purpose of this paper is to explore the neglected subtopic of probeable generalization measures; to establish firm ground for further investigations, and to inspire and guide the development of novel model tuning and optimization algorithms. We evaluate and compare measures, demonstrating effectiveness and robustness across model variations, dataset complexities, training hyperparameters, and training stages. We also introduce a new dataset of trained models and performance metrics, GenProb, for testing generalization measures, model tuning algorithms and optimization algorithms.","url_abs":"https://arxiv.org/abs/2110.12259v1","url_pdf":"https://arxiv.org/pdf/2110.12259v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"in-search-of-probeable-generalization","repo_url":"https://github.com/mahdihosseini/genprob","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2110.12259","atlas_url":"https://app.syntology.ai/?focus=2110.12259","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2110.12259"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/mahdihosseini/genprob","reach":null}],"summary":{"ran_draft_wrong":1,"ran_violates":3,"ran_fixture":1,"unverified":1},"by_repo_kind":{"official":{"samples":6,"ran":5,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"b23cbcd80c7a4289","entry":"EVBMF","repo":"mahdihosseini/genprob","repo_kind":"official","path":"source/process.py","file_url":"https://github.com/mahdihosseini/genprob/blob/HEAD/source/process.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b23cbcd80c7a4289"}},{"code_sha256_prefix":"e2f77eaf75568aa2","entry":"EVBsigma2","repo":"mahdihosseini/genprob","repo_kind":"official","path":"source/process.py","file_url":"https://github.com/mahdihosseini/genprob/blob/HEAD/source/process.py","link_basis":"first_harvest_node","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e2f77eaf75568aa2"}},{"code_sha256_prefix":"6627b133bdc36d11","entry":"agg","repo":"mahdihosseini/genprob","repo_kind":"official","path":"source/qualities.py","file_url":"https://github.com/mahdihosseini/genprob/blob/HEAD/source/qualities.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"6627b133bdc36d11"}},{"code_sha256_prefix":"5d48a562d0dd2843","entry":"all_aggs","repo":"mahdihosseini/genprob","repo_kind":"official","path":"source/qualities.py","file_url":"https://github.com/mahdihosseini/genprob/blob/HEAD/source/qualities.py","link_basis":"first_harvest_node","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"5d48a562d0dd2843"}},{"code_sha256_prefix":"2760d409b48cf23f","entry":"phi0","repo":"mahdihosseini/genprob","repo_kind":"official","path":"source/process.py","file_url":"https://github.com/mahdihosseini/genprob/blob/HEAD/source/process.py","link_basis":"first_harvest_node","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"2760d409b48cf23f"}},{"code_sha256_prefix":"90be6d8dd0c888ca","entry":"sqrtlog","repo":"mahdihosseini/genprob","repo_kind":"official","path":"source/qualities.py","file_url":"https://github.com/mahdihosseini/genprob/blob/HEAD/source/qualities.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"90be6d8dd0c888ca"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}