{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/deep-architecture-connectivity-matters-for","title":"Deep Architecture Connectivity Matters for Its Convergence: A Fine-Grained Analysis","arxiv_id":"2205.05662","date":"2022-05-11","proceeding":null,"authors":["Wuyang Chen","Wei Huang","Xinyu Gong","Boris Hanin","Zhangyang Wang"],"abstract":"Advanced deep neural networks (DNNs), designed by either human or AutoML algorithms, are growing increasingly complex. Diverse operations are connected by complicated connectivity patterns, e.g., various types of skip connections. Those topological compositions are empirically effective and observed to smooth the loss landscape and facilitate the gradient flow in general. However, it remains elusive to derive any principled understanding of their effects on the DNN capacity or trainability, and to understand why or in which aspect one specific connectivity pattern is better than another. In this work, we theoretically characterize the impact of connectivity patterns on the convergence of DNNs under gradient descent training in fine granularity. By analyzing a wide network's Neural Network Gaussian Process (NNGP), we are able to depict how the spectrum of an NNGP kernel propagates through a particular connectivity pattern, and how that affects the bound of convergence rates. As one practical implication of our results, we show that by a simple filtration on \"unpromising\" connectivity patterns, we can trim down the number of models to evaluate, and significantly accelerate the large-scale neural architecture search without any overhead. Code is available at: https://github.com/VITA-Group/architecture_convergence.","url_abs":"https://arxiv.org/abs/2205.05662v2","url_pdf":"https://arxiv.org/pdf/2205.05662v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"deep-architecture-connectivity-matters-for","repo_url":"https://github.com/chenwydj/architecture_convergence","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"deep-architecture-connectivity-matters-for","repo_url":"https://github.com/vita-group/architecture_convergence","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"automl","task_name":"AutoML"},{"task_slug":"architecture-search","task_name":"Neural Architecture Search"}],"methods":[{"method_slug":"gaussian-process","method_name":"Gaussian Process"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2205.05662","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2205.05662"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/VITA-Group/architecture_convergence","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/chenwydj/architecture_convergence","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/vita-group/architecture_convergence","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran_draft_wrong":3,"ran_violates":2,"ran":1,"unverified":15},"by_repo_kind":{"official":{"samples":21,"ran":6,"repositories":2}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"ad0636a128295d5a","entry":"calculate_md5","repo":"chenwydj/architecture_convergence","repo_kind":"official","path":"nas-without-training/datasets/DownsampledImageNet.py","file_url":"https://github.com/chenwydj/architecture_convergence/blob/HEAD/nas-without-training/datasets/DownsampledImageNet.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":2,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ad0636a128295d5a"}},{"code_sha256_prefix":"380a00b777aad655","entry":"check_md5","repo":"chenwydj/architecture_convergence","repo_kind":"official","path":"nas-without-training/datasets/DownsampledImageNet.py","file_url":"https://github.com/chenwydj/architecture_convergence/blob/HEAD/nas-without-training/datasets/DownsampledImageNet.py","link_basis":"harvester_set","language":"python","status":"ran_violates","verification_level":2,"contract_check":"VIOLATES","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"380a00b777aad655"}},{"code_sha256_prefix":"e947a6effcaaf32b","entry":"check_integrity","repo":"chenwydj/architecture_convergence","repo_kind":"official","path":"nas-without-training/datasets/DownsampledImageNet.py","file_url":"https://github.com/chenwydj/architecture_convergence/blob/HEAD/nas-without-training/datasets/DownsampledImageNet.py","link_basis":"harvester_set","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e947a6effcaaf32b"}},{"code_sha256_prefix":"8d7ec010c29e813b","entry":"format_time","repo":"chenwydj/architecture_convergence","repo_kind":"official","path":"mlp_code/utils.py","file_url":"https://github.com/chenwydj/architecture_convergence/blob/HEAD/mlp_code/utils.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"8d7ec010c29e813b"}},{"code_sha256_prefix":"5aa7497aa00af169","entry":"get_flop_loss","repo":"VITA-Group/architecture_convergence","repo_kind":"official","path":"nas-without-training/autodl/procedures/search_main.py","file_url":"https://github.com/VITA-Group/architecture_convergence/blob/HEAD/nas-without-training/autodl/procedures/search_main.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"5aa7497aa00af169"}},{"code_sha256_prefix":"3cbdd3b190b16350","entry":"get_mean_and_std","repo":"chenwydj/architecture_convergence","repo_kind":"official","path":"mlp_code/utils.py","file_url":"https://github.com/chenwydj/architecture_convergence/blob/HEAD/mlp_code/utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"3cbdd3b190b16350"}},{"code_sha256_prefix":"fce4d8316ff3d185","entry":"accuracy","repo":"chenwydj/architecture_convergence","repo_kind":"official","path":"mlp_code/utils.py","file_url":"https://github.com/chenwydj/architecture_convergence/blob/HEAD/mlp_code/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"fce4d8316ff3d185"}},{"code_sha256_prefix":"d9b7f522f765561e","entry":"edge2value","repo":"chenwydj/architecture_convergence","repo_kind":"official","path":"nas-without-training/collect_arch_data.py","file_url":"https://github.com/chenwydj/architecture_convergence/blob/HEAD/nas-without-training/collect_arch_data.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d9b7f522f765561e"}},{"code_sha256_prefix":"c2c1522bfc9c4a8b","entry":"find_all_paths","repo":"chenwydj/architecture_convergence","repo_kind":"official","path":"nas-without-training/collect_arch_data.py","file_url":"https://github.com/chenwydj/architecture_convergence/blob/HEAD/nas-without-training/collect_arch_data.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c2c1522bfc9c4a8b"}},{"code_sha256_prefix":"5cc4e76addc371ea","entry":"find_all_paths_3d_prob","repo":"chenwydj/architecture_convergence","repo_kind":"official","path":"TENAS/prune_tenas.py","file_url":"https://github.com/chenwydj/architecture_convergence/blob/HEAD/TENAS/prune_tenas.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"5cc4e76addc371ea"}},{"code_sha256_prefix":"e7c03b50082d9c61","entry":"get_batch_jacobian","repo":"chenwydj/architecture_convergence","repo_kind":"official","path":"nas-without-training/score_networks.py","file_url":"https://github.com/chenwydj/architecture_convergence/blob/HEAD/nas-without-training/score_networks.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e7c03b50082d9c61"}},{"code_sha256_prefix":"44f160db0172cb70","entry":"get_cifar_test_loader","repo":"chenwydj/architecture_convergence","repo_kind":"official","path":"mlp_code/dataset.py","file_url":"https://github.com/chenwydj/architecture_convergence/blob/HEAD/mlp_code/dataset.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"44f160db0172cb70"}},{"code_sha256_prefix":"16139ec12e4595a3","entry":"get_score_func","repo":"chenwydj/architecture_convergence","repo_kind":"official","path":"nas-without-training/scores.py","file_url":"https://github.com/chenwydj/architecture_convergence/blob/HEAD/nas-without-training/scores.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"16139ec12e4595a3"}},{"code_sha256_prefix":"aa037b9bfa339105","entry":"get_svhn_test_loader","repo":"chenwydj/architecture_convergence","repo_kind":"official","path":"mlp_code/dataset.py","file_url":"https://github.com/chenwydj/architecture_convergence/blob/HEAD/mlp_code/dataset.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"aa037b9bfa339105"}},{"code_sha256_prefix":"8d9701991ac208d4","entry":"hooklogdet","repo":"chenwydj/architecture_convergence","repo_kind":"official","path":"nas-without-training/scores.py","file_url":"https://github.com/chenwydj/architecture_convergence/blob/HEAD/nas-without-training/scores.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"8d9701991ac208d4"}},{"code_sha256_prefix":"e1591bd29b12e21d","entry":"init_model","repo":"chenwydj/architecture_convergence","repo_kind":"official","path":"TENAS/prune_tenas.py","file_url":"https://github.com/chenwydj/architecture_convergence/blob/HEAD/TENAS/prune_tenas.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e1591bd29b12e21d"}},{"code_sha256_prefix":"2cd6cda6c652de6d","entry":"make_colours_by_hue","repo":"chenwydj/architecture_convergence","repo_kind":"official","path":"nas-without-training/plot_scores.py","file_url":"https://github.com/chenwydj/architecture_convergence/blob/HEAD/nas-without-training/plot_scores.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"2cd6cda6c652de6d"}},{"code_sha256_prefix":"a87bbc139870cdc0","entry":"prepare_logger","repo":"chenwydj/architecture_convergence","repo_kind":"official","path":"mlp_code/logger.py","file_url":"https://github.com/chenwydj/architecture_convergence/blob/HEAD/mlp_code/logger.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a87bbc139870cdc0"}},{"code_sha256_prefix":"3aa2047d05e1a518","entry":"random_score","repo":"chenwydj/architecture_convergence","repo_kind":"official","path":"nas-without-training/scores.py","file_url":"https://github.com/chenwydj/architecture_convergence/blob/HEAD/nas-without-training/scores.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"3aa2047d05e1a518"}},{"code_sha256_prefix":"65d81629d7179cf5","entry":"round_to","repo":"chenwydj/architecture_convergence","repo_kind":"official","path":"TENAS/prune_tenas.py","file_url":"https://github.com/chenwydj/architecture_convergence/blob/HEAD/TENAS/prune_tenas.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"65d81629d7179cf5"}},{"code_sha256_prefix":"cc11bb618b200b77","entry":"svhn_dataloaders","repo":"chenwydj/architecture_convergence","repo_kind":"official","path":"mlp_code/dataset.py","file_url":"https://github.com/chenwydj/architecture_convergence/blob/HEAD/mlp_code/dataset.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"cc11bb618b200b77"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}