{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/layer-folding-neural-network-depth-reduction","title":"Layer Folding: Neural Network Depth Reduction using Activation Linearization","arxiv_id":"2106.09309","date":"2021-06-17","proceeding":null,"authors":["Amir Ben Dror","Niv Zehngut","Avraham Raviv","Evgeny Artyomov","Ran Vitek","Roy Jevnisek"],"abstract":"Despite the increasing prevalence of deep neural networks, their applicability in resource-constrained devices is limited due to their computational load. While modern devices exhibit a high level of parallelism, real-time latency is still highly dependent on networks' depth. Although recent works show that below a certain depth, the width of shallower networks must grow exponentially, we presume that neural networks typically exceed this minimal depth to accelerate convergence and incrementally increase accuracy. This motivates us to transform pre-trained deep networks that already exploit such advantages into shallower forms. We propose a method that learns whether non-linear activations can be removed, allowing to fold consecutive linear layers into one. We apply our method to networks pre-trained on CIFAR-10 and CIFAR-100 and find that they can all be transformed into shallower forms that share a similar depth. Finally, we use our method to provide more efficient alternatives to MobileNetV2 and EfficientNet-Lite architectures on the ImageNet classification task.","url_abs":"https://arxiv.org/abs/2106.09309v2","url_pdf":"https://arxiv.org/pdf/2106.09309v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"layer-folding-neural-network-depth-reduction","repo_url":"https://github.com/LayerFolding/Layer-Folding","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"BSD-3-Clause"}}],"tasks":[],"methods":[{"method_slug":"1x1-convolution","method_name":"1x1 Convolution"},{"method_slug":"average-pooling","method_name":"Average Pooling"},{"method_slug":"batch-normalization","method_name":"Batch Normalization"},{"method_slug":"convolution","method_name":"Convolution"},{"method_slug":"depthwise-convolution","method_name":"Depthwise Convolution"},{"method_slug":"depthwise-separable-convolution","method_name":"Depthwise Separable Convolution"},{"method_slug":"inverted-residual-block","method_name":"Inverted Residual Block"},{"method_slug":"pointwise-convolution","method_name":"Pointwise Convolution"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2106.09309","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2106.09309"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/LayerFolding/Layer-Folding","reach":{"status":"ok","spdx":"BSD-3-Clause"}}],"summary":{"ran":1,"unverified":11},"by_repo_kind":{"official":{"samples":12,"ran":1,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"b6e06004f7efe660","entry":"wrapped_partial","repo":"LayerFolding/Layer-Folding","repo_kind":"official","path":"Folded-Mobilenets/mobilenet_v2.py","file_url":"https://github.com/LayerFolding/Layer-Folding/blob/HEAD/Folded-Mobilenets/mobilenet_v2.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"b6e06004f7efe660"}},{"code_sha256_prefix":"ebb225e100d48e3d","entry":"bytes_list_feature","repo":"LayerFolding/Layer-Folding","repo_kind":"official","path":"Folded-Mobilenets/dataset_utils.py","file_url":"https://github.com/LayerFolding/Layer-Folding/blob/HEAD/Folded-Mobilenets/dataset_utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"ebb225e100d48e3d"}},{"code_sha256_prefix":"9e4ed0ee1e56b044","entry":"conv","repo":"LayerFolding/Layer-Folding","repo_kind":"official","path":"ResNet_Cifar10_postfold.py","file_url":"https://github.com/LayerFolding/Layer-Folding/blob/HEAD/ResNet_Cifar10_postfold.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"9e4ed0ee1e56b044"}},{"code_sha256_prefix":"09dcc50c884b217d","entry":"conv5x5","repo":"LayerFolding/Layer-Folding","repo_kind":"official","path":"VGG_Cifar10_postfold.py","file_url":"https://github.com/LayerFolding/Layer-Folding/blob/HEAD/VGG_Cifar10_postfold.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"09dcc50c884b217d"}},{"code_sha256_prefix":"2bd76ec2dd72540c","entry":"expand_input_by_factor","repo":"LayerFolding/Layer-Folding","repo_kind":"official","path":"Folded-Mobilenets/conv_blocks.py","file_url":"https://github.com/LayerFolding/Layer-Folding/blob/HEAD/Folded-Mobilenets/conv_blocks.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"2bd76ec2dd72540c"}},{"code_sha256_prefix":"a937f6f8be8fc311","entry":"float_list_feature","repo":"LayerFolding/Layer-Folding","repo_kind":"official","path":"Folded-Mobilenets/dataset_utils.py","file_url":"https://github.com/LayerFolding/Layer-Folding/blob/HEAD/Folded-Mobilenets/dataset_utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"a937f6f8be8fc311"}},{"code_sha256_prefix":"537870c8bbcd9607","entry":"int64_feature","repo":"LayerFolding/Layer-Folding","repo_kind":"official","path":"Folded-Mobilenets/dataset_utils.py","file_url":"https://github.com/LayerFolding/Layer-Folding/blob/HEAD/Folded-Mobilenets/dataset_utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"537870c8bbcd9607"}},{"code_sha256_prefix":"718e3968934c45b7","entry":"make_layers","repo":"LayerFolding/Layer-Folding","repo_kind":"official","path":"VGG_Cifar10_postfold.py","file_url":"https://github.com/LayerFolding/Layer-Folding/blob/HEAD/VGG_Cifar10_postfold.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"718e3968934c45b7"}},{"code_sha256_prefix":"3ef89f5a98a3352d","entry":"preprocess_image","repo":"LayerFolding/Layer-Folding","repo_kind":"official","path":"Folded-Mobilenets/preprocessing.py","file_url":"https://github.com/LayerFolding/Layer-Folding/blob/HEAD/Folded-Mobilenets/preprocessing.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"3ef89f5a98a3352d"}},{"code_sha256_prefix":"58bbc7ac2c2a5474","entry":"vgg16","repo":"LayerFolding/Layer-Folding","repo_kind":"official","path":"VGG_Cifar10_prefold.py","file_url":"https://github.com/LayerFolding/Layer-Folding/blob/HEAD/VGG_Cifar10_prefold.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"58bbc7ac2c2a5474"}},{"code_sha256_prefix":"1ad77b38966daedd","entry":"vgg19","repo":"LayerFolding/Layer-Folding","repo_kind":"official","path":"VGG_Cifar10_postfold.py","file_url":"https://github.com/LayerFolding/Layer-Folding/blob/HEAD/VGG_Cifar10_postfold.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"1ad77b38966daedd"}},{"code_sha256_prefix":"50714c640ea17fb6","entry":"vgg19","repo":"LayerFolding/Layer-Folding","repo_kind":"official","path":"VGG_Cifar10_prefold.py","file_url":"https://github.com/LayerFolding/Layer-Folding/blob/HEAD/VGG_Cifar10_prefold.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"50714c640ea17fb6"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}