{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/pretraining-a-neural-network-before-knowing","title":"Pretraining a Neural Network before Knowing Its Architecture","arxiv_id":"2207.10049","date":"2022-07-20","proceeding":null,"authors":["Boris Knyazev"],"abstract":"Training large neural networks is possible by training a smaller hypernetwork that predicts parameters for the large ones. A recently released Graph HyperNetwork (GHN) trained this way on one million smaller ImageNet architectures is able to predict parameters for large unseen networks such as ResNet-50. While networks with predicted parameters lose performance on the source task, the predicted parameters have been found useful for fine-tuning on other tasks. We study if fine-tuning based on the same GHN is still useful on novel strong architectures that were published after the GHN had been trained. We found that for recent architectures such as ConvNeXt, GHN initialization becomes less useful than for ResNet-50. One potential reason is the increased distribution shift of novel architectures from those used to train the GHN. We also found that the predicted parameters lack the diversity necessary to successfully fine-tune parameters with gradient descent. We alleviate this limitation by applying simple post-processing techniques to predicted parameters before fine-tuning them on a target task and improve fine-tuning of ResNet-50 and ConvNeXt.","url_abs":"https://arxiv.org/abs/2207.10049v1","url_pdf":"https://arxiv.org/pdf/2207.10049v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"pretraining-a-neural-network-before-knowing","repo_url":"https://github.com/facebookresearch/ppuda","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"pretraining-a-neural-network-before-knowing","repo_url":"https://github.com/yaya-yns/tart","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"diversity","task_name":"Diversity"}],"methods":[{"method_slug":"convnext","method_name":"ConvNeXt"},{"method_slug":"hypernetwork","method_name":"HyperNetwork"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2207.10049","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2207.10049"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/facebookresearch/ppuda","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/yaya-yns/tart","reach":null}],"summary":{"ran_draft_wrong":2,"ran_fixture":1,"unverified":1},"by_repo_kind":{"official":{"samples":1,"ran":0,"repositories":1},"listed":{"samples":3,"ran":3,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"e5789243721bd286","entry":"graph_pooling","repo":"yaya-yns/tart","repo_kind":"listed","path":"experiments/neural_predictor.py","file_url":"https://github.com/yaya-yns/tart/blob/HEAD/experiments/neural_predictor.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e5789243721bd286"}},{"code_sha256_prefix":"dfa62dfebede20fc","entry":"normalize_adj","repo":"yaya-yns/tart","repo_kind":"listed","path":"experiments/neural_predictor.py","file_url":"https://github.com/yaya-yns/tart/blob/HEAD/experiments/neural_predictor.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"dfa62dfebede20fc"}},{"code_sha256_prefix":"54e37c22f817d98b","entry":"to_cuda","repo":"yaya-yns/tart","repo_kind":"listed","path":"experiments/neural_predictor.py","file_url":"https://github.com/yaya-yns/tart/blob/HEAD/experiments/neural_predictor.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"54e37c22f817d98b"}},{"code_sha256_prefix":"51210ced01277bb6","entry":"ghn_parallel","repo":"facebookresearch/ppuda","repo_kind":"official","path":"ppuda/ghn/nn.py","file_url":"https://github.com/facebookresearch/ppuda/blob/HEAD/ppuda/ghn/nn.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"51210ced01277bb6"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}