{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/evolving-normalization-activation-layers","title":"Evolving Normalization-Activation Layers","arxiv_id":"2004.02967","date":"2020-04-06","proceeding":"NeurIPS 2020 12","authors":["Hanxiao Liu","Andrew Brock","Karen Simonyan","Quoc V. Le"],"abstract":"Normalization layers and activation functions are fundamental components in deep networks and typically co-locate with each other. Here we propose to design them using an automated approach. Instead of designing them separately, we unify them into a single tensor-to-tensor computation graph, and evolve its structure starting from basic mathematical functions. Examples of such mathematical functions are addition, multiplication and statistical moments. The use of low-level mathematical functions, in contrast to the use of high-level modules in mainstream NAS, leads to a highly sparse and large search space which can be challenging for search methods. To address the challenge, we develop efficient rejection protocols to quickly filter out candidate layers that do not work well. We also use multi-objective evolution to optimize each layer's performance across many architectures to prevent overfitting. Our method leads to the discovery of EvoNorms, a set of new normalization-activation layers with novel, and sometimes surprising structures that go beyond existing design patterns. For example, some EvoNorms do not assume that normalization and activation functions must be applied sequentially, nor need to center the feature maps, nor require explicit activation functions. Our experiments show that EvoNorms work well on image classification models including ResNets, MobileNets and EfficientNets but also transfer well to Mask R-CNN with FPN/SpineNet for instance segmentation and to BigGAN for image synthesis, outperforming BatchNorm and GroupNorm based layers in many cases.","url_abs":"https://arxiv.org/abs/2004.02967v5","url_pdf":"https://arxiv.org/pdf/2004.02967v5.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"evolving-normalization-activation-layers","repo_url":"https://github.com/dinrker/EvoNorms-CIFAR","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}},{"paper_slug":"evolving-normalization-activation-layers","repo_url":"https://github.com/lonePatient/EvoNorms_PyTorch","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"evolving-normalization-activation-layers","repo_url":"https://github.com/mnikitin/EvoNorm","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"mxnet","reach":{"status":"ok"}},{"paper_slug":"evolving-normalization-activation-layers","repo_url":"https://github.com/sayakpaul/EvoNorms-in-TensorFlow-2","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"tf","reach":{"status":"ok"}},{"paper_slug":"evolving-normalization-activation-layers","repo_url":"https://github.com/wandb/awesome-dl-projects","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"tf","reach":{"status":"ok"}},{"paper_slug":"evolving-normalization-activation-layers","repo_url":"https://github.com/digantamisra98/EvoNorm","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"pytorch","reach":null},{"paper_slug":"evolving-normalization-activation-layers","repo_url":"https://github.com/leondgarse/keras_cv_attention_models/tree/main/keras_cv_attention_models/resnet_family","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"tf","reach":null},{"paper_slug":"evolving-normalization-activation-layers","repo_url":"https://github.com/rwightman/pytorch-image-models","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"image-classification","task_name":"Image Classification"},{"task_slug":"image-generation","task_name":"Image Generation"},{"task_slug":"instance-segmentation","task_name":"Instance Segmentation"},{"task_slug":"semantic-segmentation","task_name":"Semantic Segmentation"},{"task_slug":"image-classification","task_name":"image-classification"}],"methods":[{"method_slug":"1x1-convolution","method_name":"1x1 Convolution"},{"method_slug":"adam","method_name":"Adam"},{"method_slug":"average-pooling","method_name":"Average Pooling"},{"method_slug":"batch-normalization","method_name":"Batch Normalization"},{"method_slug":"biggan","method_name":"BigGAN"},{"method_slug":"bottleneck-residual-block","method_name":"Bottleneck Residual Block"},{"method_slug":"conditional-batch-normalization","method_name":"Conditional Batch Normalization"},{"method_slug":"convolution","method_name":"Convolution"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"depthwise-convolution","method_name":"Depthwise Convolution"},{"method_slug":"depthwise-separable-convolution","method_name":"Depthwise Separable Convolution"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"early-stopping","method_name":"Early Stopping"},{"method_slug":"efficientnet","method_name":"EfficientNet"},{"method_slug":"entropy-regularization","method_name":"Entropy Regularization"},{"method_slug":"evonorms","method_name":"EvoNorms"},{"method_slug":"fpn","method_name":"FPN"},{"method_slug":"feedforward-network","method_name":"Feedforward Network"},{"method_slug":"gan-hinge-loss","method_name":"GAN Hinge Loss"},{"method_slug":"global-average-pooling","method_name":"Global Average Pooling"},{"method_slug":"group-normalization","method_name":"Group Normalization"},{"method_slug":"inverted-residual-block","method_name":"Inverted Residual Block"},{"method_slug":"kaiming-initialization","method_name":"Kaiming Initialization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"mask-r-cnn","method_name":"Mask R-CNN"},{"method_slug":"max-pooling","method_name":"Max Pooling"},{"method_slug":"non-local-block","method_name":"Non-Local Block"},{"method_slug":"non-local-operation","method_name":"Non-Local Operation"},{"method_slug":"off-diagonal-orthogonal-regularization","method_name":"Off-Diagonal Orthogonal Regularization"},{"method_slug":"pointwise-convolution","method_name":"Pointwise Convolution"},{"method_slug":"projection-discriminator","method_name":"Projection Discriminator"},{"method_slug":"rmsprop","method_name":"RMSProp"},{"method_slug":"rpn","method_name":"RPN"},{"method_slug":"relu","method_name":"ReLU"},{"method_slug":"residual-block","method_name":"Residual Block"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"roi-align","method_name":"RoIAlign"},{"method_slug":"sagan","method_name":"SAGAN"},{"method_slug":"sigmoid-activation","method_name":"Sigmoid Activation"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"spectral-normalization","method_name":"Spectral Normalization"},{"method_slug":"spinenet","method_name":"SpineNet"},{"method_slug":"squeeze-and-excitation-block","method_name":"Squeeze-and-Excitation Block"},{"method_slug":"ttur","method_name":"TTUR"},{"method_slug":"tanh-activation","method_name":"Tanh Activation"},{"method_slug":"truncation-trick","method_name":"Truncation Trick"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2004.02967","atlas_url":"https://app.syntology.ai/?focus=2004.02967","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2004.02967"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/rwightman/pytorch-image-models","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/sayakpaul/EvoNorms-in-TensorFlow-2","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/lonePatient/EvoNorms_PyTorch","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/leondgarse/keras_cv_attention_models/tree/main/keras_cv_attention_models/resnet_family","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/digantamisra98/EvoNorm","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/dinrker/EvoNorms-CIFAR","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/mnikitin/EvoNorm","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/wandb/awesome-dl-projects","reach":{"status":"ok"}}],"summary":{"ran_draft_wrong":3,"unverified":2},"by_repo_kind":{"listed":{"samples":4,"ran":3,"repositories":2}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":3,"samples":[{"code_sha256_prefix":"8919f8f4b59222a5","entry":"group_std","repo":"lonePatient/EvoNorms_PyTorch","repo_kind":"listed","path":"models/normalization.py","file_url":"https://github.com/lonePatient/EvoNorms_PyTorch/blob/HEAD/models/normalization.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"8919f8f4b59222a5"}},{"code_sha256_prefix":"09b17e7339f8c96a","entry":"instance_std","repo":"lonePatient/EvoNorms_PyTorch","repo_kind":"listed","path":"models/normalization.py","file_url":"https://github.com/lonePatient/EvoNorms_PyTorch/blob/HEAD/models/normalization.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"09b17e7339f8c96a"}},{"code_sha256_prefix":"041f37e1f42544de","entry":"instance_std","repo":"digantamisra98/EvoNorm","repo_kind":"listed","path":"models/evonorm2d.py","file_url":"https://github.com/digantamisra98/EvoNorm/blob/HEAD/models/evonorm2d.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"041f37e1f42544de"}},{"code_sha256_prefix":"f0c9a29156911331","entry":"accuracy","repo":null,"repo_kind":null,"path":null,"file_url":null,"link_basis":"identical_code_first_harvested_elsewhere","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":null,"inline_ok":false,"mcp_get_code":{"code_sha256":"f0c9a29156911331"}},{"code_sha256_prefix":"f2f879b4407d0abc","entry":"group_std","repo":"digantamisra98/EvoNorm","repo_kind":"listed","path":"models/evonorm2d.py","file_url":"https://github.com/digantamisra98/EvoNorm/blob/HEAD/models/evonorm2d.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f2f879b4407d0abc"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}