{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/gradsign-model-performance-inference-with-1","title":"GradSign: Model Performance Inference with Theoretical Insights","arxiv_id":"2110.08616","date":"2021-10-16","proceeding":"ICLR 2022 4","authors":["Zhihao Zhang","Zhihao Jia"],"abstract":"A key challenge in neural architecture search (NAS) is quickly inferring the predictive performance of a broad spectrum of networks to discover statistically accurate and computationally efficient ones. We refer to this task as model performance inference (MPI). The current practice for efficient MPI is gradient-based methods that leverage the gradients of a network at initialization to infer its performance. However, existing gradient-based methods rely only on heuristic metrics and lack the necessary theoretical foundations to consolidate their designs. We propose GradSign, an accurate, simple, and flexible metric for model performance inference with theoretical insights. The key idea behind GradSign is a quantity {\\Psi} to analyze the optimization landscape of different networks at the granularity of individual training samples. Theoretically, we show that both the network's training and true population losses are proportionally upper-bounded by {\\Psi} under reasonable assumptions. In addition, we design GradSign, an accurate and simple approximation of {\\Psi} using the gradients of a network evaluated at a random initialization state. Evaluation on seven NAS benchmarks across three training datasets shows that GradSign generalizes well to real-world networks and consistently outperforms state-of-the-art gradient-based methods for MPI evaluated by Spearman's {\\rho} and Kendall's Tau. Additionally, we integrate GradSign into four existing NAS algorithms and show that the GradSign-assisted NAS algorithms outperform their vanilla counterparts by improving the accuracies of best-discovered networks by up to 0.3%, 1.1%, and 1.0% on three real-world tasks.","url_abs":"https://arxiv.org/abs/2110.08616v2","url_pdf":"https://arxiv.org/pdf/2110.08616v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"gradsign-model-performance-inference-with-1","repo_url":"https://github.com/cmu-catalyst/gradsign","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"architecture-search","task_name":"Neural Architecture Search"},{"task_slug":"model","task_name":"model"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2110.08616","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2110.08616"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/cmu-catalyst/gradsign","reach":null}],"summary":{"ran_honours":4,"ran_draft_wrong":1},"by_repo_kind":{"official":{"samples":5,"ran":5,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":5,"samples":[{"code_sha256_prefix":"020af7f72a6bb192","entry":"get_flattened_metric","repo":"cmu-catalyst/gradsign","repo_kind":"official","path":"naswot-code/gradsign.py","file_url":"https://github.com/cmu-catalyst/gradsign/blob/HEAD/naswot-code/gradsign.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"020af7f72a6bb192"}},{"code_sha256_prefix":"537831f0e0f67205","entry":"get_grad_conflict","repo":"cmu-catalyst/gradsign","repo_kind":"official","path":"naswot-code/gradsign.py","file_url":"https://github.com/cmu-catalyst/gradsign/blob/HEAD/naswot-code/gradsign.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"537831f0e0f67205"}},{"code_sha256_prefix":"94369cfdcbec594e","entry":"get_layer_metric_array","repo":"cmu-catalyst/gradsign","repo_kind":"official","path":"naswot-code/gradsign.py","file_url":"https://github.com/cmu-catalyst/gradsign/blob/HEAD/naswot-code/gradsign.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"94369cfdcbec594e"}},{"code_sha256_prefix":"dbc9b4d67b87c439","entry":"get_num_classes","repo":"cmu-catalyst/gradsign","repo_kind":"official","path":"zero-cost-nas-code/build.py","file_url":"https://github.com/cmu-catalyst/gradsign/blob/HEAD/zero-cost-nas-code/build.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"dbc9b4d67b87c439"}},{"code_sha256_prefix":"39e4146c98293a4d","entry":"sum_arr","repo":"cmu-catalyst/gradsign","repo_kind":"official","path":"naswot-code/gradsign.py","file_url":"https://github.com/cmu-catalyst/gradsign/blob/HEAD/naswot-code/gradsign.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"39e4146c98293a4d"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}