{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/continuous-in-depth-neural-networks","title":"Continuous-in-Depth Neural Networks","arxiv_id":"2008.02389","date":"2020-08-05","proceeding":null,"authors":["Alejandro F. Queiruga","N. Benjamin Erichson","Dane Taylor","Michael W. Mahoney"],"abstract":"Recent work has attempted to interpret residual networks (ResNets) as one step of a forward Euler discretization of an ordinary differential equation, focusing mainly on syntactic algebraic similarities between the two systems. Discrete dynamical integrators of continuous dynamical systems, however, have a much richer structure. We first show that ResNets fail to be meaningful dynamical integrators in this richer sense. We then demonstrate that neural network models can learn to represent continuous dynamical systems, with this richer structure and properties, by embedding them into higher-order numerical integration schemes, such as the Runge Kutta schemes. Based on these insights, we introduce ContinuousNet as a continuous-in-depth generalization of ResNet architectures. ContinuousNets exhibit an invariance to the particular computational graph manifestation. That is, the continuous-in-depth model can be evaluated with different discrete time step sizes, which changes the number of layers, and different numerical integration schemes, which changes the graph connectivity. We show that this can be used to develop an incremental-in-depth training scheme that improves model quality, while significantly decreasing training time. We also show that, once trained, the number of units in the computational graph can even be decreased, for faster inference with little-to-no accuracy drop.","url_abs":"https://arxiv.org/abs/2008.02389v1","url_pdf":"https://arxiv.org/pdf/2008.02389v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"continuous-in-depth-neural-networks","repo_url":"https://github.com/afqueiruga/ContinuousNet","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"GPL-3.0"}},{"paper_slug":"continuous-in-depth-neural-networks","repo_url":"https://github.com/afqueiruga/ODENet","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"GPL-3.0"}},{"paper_slug":"continuous-in-depth-neural-networks","repo_url":"https://github.com/afqueiruga/StatefulOdeNets","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"jax","reach":null},{"paper_slug":"continuous-in-depth-neural-networks","repo_url":"https://github.com/ivandariojr/lyapunovlearning","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"numerical-integration","task_name":"Numerical Integration"}],"methods":[{"method_slug":"1x1-convolution","method_name":"1x1 Convolution"},{"method_slug":"average-pooling","method_name":"Average Pooling"},{"method_slug":"batch-normalization","method_name":"Batch Normalization"},{"method_slug":"bottleneck-residual-block","method_name":"Bottleneck Residual Block"},{"method_slug":"convolution","method_name":"Convolution"},{"method_slug":"global-average-pooling","method_name":"Global Average Pooling"},{"method_slug":"kaiming-initialization","method_name":"Kaiming Initialization"},{"method_slug":"max-pooling","method_name":"Max Pooling"},{"method_slug":"relu","method_name":"ReLU"},{"method_slug":"residual-block","method_name":"Residual Block"},{"method_slug":"residual-connection","method_name":"Residual Connection"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2008.02389","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2008.02389"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/afqueiruga/StatefulOdeNets","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/afqueiruga/ODENet","reach":{"status":"ok","spdx":"GPL-3.0"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/afqueiruga/ContinuousNet","reach":{"status":"ok","spdx":"GPL-3.0"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/ivandariojr/lyapunovlearning","reach":null}],"summary":{"ran_draft_wrong":4,"unverified":2},"by_repo_kind":{"listed":{"samples":6,"ran":4,"repositories":2}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":6,"samples":[{"code_sha256_prefix":"f3ee1c97f81b2274","entry":"get_arch_name","repo":"ivandariojr/lyapunovlearning","repo_kind":"listed","path":"sl_pipeline.py","file_url":"https://github.com/ivandariojr/lyapunovlearning/blob/HEAD/sl_pipeline.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"f3ee1c97f81b2274"}},{"code_sha256_prefix":"2f26b981f0269a1c","entry":"get_model_name","repo":"ivandariojr/lyapunovlearning","repo_kind":"listed","path":"sl_pipeline.py","file_url":"https://github.com/ivandariojr/lyapunovlearning/blob/HEAD/sl_pipeline.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"2f26b981f0269a1c"}},{"code_sha256_prefix":"9501e3c98d3be436","entry":"get_module_name","repo":"ivandariojr/lyapunovlearning","repo_kind":"listed","path":"sl_pipeline.py","file_url":"https://github.com/ivandariojr/lyapunovlearning/blob/HEAD/sl_pipeline.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"9501e3c98d3be436"}},{"code_sha256_prefix":"7634096bbbbbf744","entry":"initialize_multiple_times","repo":"afqueiruga/StatefulOdeNets","repo_kind":"listed","path":"continuous_net/continuous_block.py","file_url":"https://github.com/afqueiruga/StatefulOdeNets/blob/HEAD/continuous_net/continuous_block.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"GPL-3.0","inline_ok":false,"mcp_get_code":{"code_sha256":"7634096bbbbbf744"}},{"code_sha256_prefix":"c0ff2d556897a83c","entry":"copy_and_perturb","repo":"afqueiruga/StatefulOdeNets","repo_kind":"listed","path":"continuous_net/continuous_block.py","file_url":"https://github.com/afqueiruga/StatefulOdeNets/blob/HEAD/continuous_net/continuous_block.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"GPL-3.0","inline_ok":false,"mcp_get_code":{"code_sha256":"c0ff2d556897a83c"}},{"code_sha256_prefix":"4df980ad796e5690","entry":"initialize_multiple_times_split_state","repo":"afqueiruga/StatefulOdeNets","repo_kind":"listed","path":"continuous_net/continuous_block.py","file_url":"https://github.com/afqueiruga/StatefulOdeNets/blob/HEAD/continuous_net/continuous_block.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"GPL-3.0","inline_ok":false,"mcp_get_code":{"code_sha256":"4df980ad796e5690"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}