{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/high-performance-large-scale-image","title":"High-Performance Large-Scale Image Recognition Without Normalization","arxiv_id":"2102.06171","date":"2021-02-11","proceeding":null,"authors":["Andrew Brock","Soham De","Samuel L. Smith","Karen Simonyan"],"abstract":"Batch normalization is a key component of most image classification models, but it has many undesirable properties stemming from its dependence on the batch size and interactions between examples. Although recent work has succeeded in training deep ResNets without normalization layers, these models do not match the test accuracies of the best batch-normalized networks, and are often unstable for large learning rates or strong data augmentations. In this work, we develop an adaptive gradient clipping technique which overcomes these instabilities, and design a significantly improved class of Normalizer-Free ResNets. Our smaller models match the test accuracy of an EfficientNet-B7 on ImageNet while being up to 8.7x faster to train, and our largest models attain a new state-of-the-art top-1 accuracy of 86.5%. In addition, Normalizer-Free models attain significantly better performance than their batch-normalized counterparts when finetuning on ImageNet after large-scale pre-training on a dataset of 300 million labeled images, with our best models obtaining an accuracy of 89.2%. Our code is available at https://github.com/deepmind/ deepmind-research/tree/master/nfnets","url_abs":"https://arxiv.org/abs/2102.06171v1","url_pdf":"https://arxiv.org/pdf/2102.06171v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"high-performance-large-scale-image","repo_url":"https://github.com/deepmind/deepmind-research","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"tf","reach":null},{"paper_slug":"high-performance-large-scale-image","repo_url":"https://github.com/deepmind/deepmind-research/tree/master/nfnets","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"jax","reach":null},{"paper_slug":"high-performance-large-scale-image","repo_url":"https://github.com/ahsanmemon/NFNets-Tensorflow","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"tf","reach":{"status":"unanswered"}},{"paper_slug":"high-performance-large-scale-image","repo_url":"https://github.com/ayulockin/Explore-NFNet","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"tf","reach":{"status":"unanswered"}},{"paper_slug":"high-performance-large-scale-image","repo_url":"https://github.com/benjs/nfnets_pytorch","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"unanswered"}},{"paper_slug":"high-performance-large-scale-image","repo_url":"https://github.com/callmefish/NFnet","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"high-performance-large-scale-image","repo_url":"https://github.com/hoangthang1607/nfnets-Tensorflow-2","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"tf","reach":{"status":"unanswered"}},{"paper_slug":"high-performance-large-scale-image","repo_url":"https://github.com/nmichlo/experiment-normalized-activations","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"unanswered"}},{"paper_slug":"high-performance-large-scale-image","repo_url":"https://github.com/psh01087/simple-nfnet","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"unanswered"}},{"paper_slug":"high-performance-large-scale-image","repo_url":"https://github.com/rwightman/efficientdet-pytorch","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"unanswered"}},{"paper_slug":"high-performance-large-scale-image","repo_url":"https://github.com/rwightman/pytorch-image-models","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"high-performance-large-scale-image","repo_url":"https://github.com/sayakpaul/Adaptive-Gradient-Clipping","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"tf","reach":null},{"paper_slug":"high-performance-large-scale-image","repo_url":"https://github.com/sense-x/tokenmix","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"high-performance-large-scale-image","repo_url":"https://github.com/smilingwolf/sw-cv-modelzoo","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"tf","reach":{"status":"unanswered"}},{"paper_slug":"high-performance-large-scale-image","repo_url":"https://github.com/vballoli/nfnets-pytorch","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"unanswered"}},{"paper_slug":"high-performance-large-scale-image","repo_url":"https://github.com/ypeleg/nfnets-keras","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"tf","reach":{"status":"unanswered"}},{"paper_slug":"high-performance-large-scale-image","repo_url":"https://github.com/MS-Mind/MS-Code-06/tree/main/NFNet","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"mindspore","reach":null},{"paper_slug":"high-performance-large-scale-image","repo_url":"https://github.com/leondgarse/keras_cv_attention_models/tree/main/keras_cv_attention_models/nfnets","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"tf","reach":null},{"paper_slug":"high-performance-large-scale-image","repo_url":"https://github.com/mindspore-ai/models/tree/master/research/cv/NFNet","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"mindspore","reach":null},{"paper_slug":"high-performance-large-scale-image","repo_url":"https://gitlab.com/birder/birder","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"image-classification","task_name":"Image Classification"},{"task_slug":"high","task_name":"Vocal Bursts Intensity Prediction"},{"task_slug":"image-classification","task_name":"image-classification"}],"methods":[{"method_slug":"gradient-clipping","method_name":"Gradient Clipping"}],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/image-classification-on-imagenet","task":"Image Classification","dataset":"ImageNet","model":"NFNet-F4+","rank_in_archive_order":23,"of":1060,"metrics":{"GFLOPs":"367","Number of params":"527M","Top 1 Accuracy":"89.2%"},"uses_additional_data":false},{"leaderboard":"/sota/image-classification-on-imagenet","task":"Image Classification","dataset":"ImageNet","model":"NFNet-F6 w/ SAM","rank_in_archive_order":141,"of":1060,"metrics":{"GFLOPs":"377.28","Number of params":"438.4M","Top 1 Accuracy":"86.5%"},"uses_additional_data":false},{"leaderboard":"/sota/image-classification-on-imagenet","task":"Image Classification","dataset":"ImageNet","model":"NFNet-F5 w/ SAM","rank_in_archive_order":163,"of":1060,"metrics":{"GFLOPs":"289.76","Number of params":"377.2M","Top 1 Accuracy":"86.3%"},"uses_additional_data":false},{"leaderboard":"/sota/image-classification-on-imagenet","task":"Image Classification","dataset":"ImageNet","model":"NFNet-F5","rank_in_archive_order":184,"of":1060,"metrics":{"GFLOPs":"289.76","Number of params":"377.2M","Top 1 Accuracy":"86.0%"},"uses_additional_data":false},{"leaderboard":"/sota/image-classification-on-imagenet","task":"Image Classification","dataset":"ImageNet","model":"NFNet-F4","rank_in_archive_order":188,"of":1060,"metrics":{"GFLOPs":"215.24","Number of params":"316.1M","Top 1 Accuracy":"85.9%"},"uses_additional_data":false},{"leaderboard":"/sota/image-classification-on-imagenet","task":"Image Classification","dataset":"ImageNet","model":"NFNet-F3","rank_in_archive_order":211,"of":1060,"metrics":{"GFLOPs":"114.76","Number of params":"254.9M","Top 1 Accuracy":"85.7%"},"uses_additional_data":false},{"leaderboard":"/sota/image-classification-on-imagenet","task":"Image Classification","dataset":"ImageNet","model":"NFNet-F2","rank_in_archive_order":260,"of":1060,"metrics":{"GFLOPs":"62.59","Number of params":"193.8M","Top 1 Accuracy":"85.1%"},"uses_additional_data":false},{"leaderboard":"/sota/image-classification-on-imagenet","task":"Image Classification","dataset":"ImageNet","model":"NFNet-F1","rank_in_archive_order":301,"of":1060,"metrics":{"GFLOPs":"35.54","Number of params":"132.6M","Top 1 Accuracy":"84.7%"},"uses_additional_data":false},{"leaderboard":"/sota/image-classification-on-imagenet","task":"Image Classification","dataset":"ImageNet","model":"NFNet-F0","rank_in_archive_order":416,"of":1060,"metrics":{"GFLOPs":"12.38","Number of params":"71.5M","Top 1 Accuracy":"83.6%"},"uses_additional_data":false}],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2102.06171","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2102.06171"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/rwightman/pytorch-image-models","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/deepmind/deepmind-research","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/rwightman/efficientdet-pytorch","reach":{"status":"unanswered"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/MS-Mind/MS-Code-06/tree/main/NFNet","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/nmichlo/experiment-normalized-activations","reach":{"status":"unanswered"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/mindspore-ai/models/tree/master/research/cv/NFNet","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/sense-x/tokenmix","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/leondgarse/keras_cv_attention_models/tree/main/keras_cv_attention_models/nfnets","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/psh01087/simple-nfnet","reach":{"status":"unanswered"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/ayulockin/Explore-NFNet","reach":{"status":"unanswered"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://gitlab.com/birder/birder","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/benjs/nfnets_pytorch","reach":{"status":"unanswered"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/sayakpaul/Adaptive-Gradient-Clipping","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/smilingwolf/sw-cv-modelzoo","reach":{"status":"unanswered"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/vballoli/nfnets-pytorch","reach":{"status":"unanswered"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/ypeleg/nfnets-keras","reach":{"status":"unanswered"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/callmefish/NFnet","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/deepmind/deepmind-research/tree/master/nfnets","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/ahsanmemon/NFNets-Tensorflow","reach":{"status":"unanswered"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/hoangthang1607/nfnets-Tensorflow-2","reach":{"status":"unanswered"}}],"summary":{"ran_fixture":3},"by_repo_kind":{"listed":{"samples":3,"ran":3,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":3,"samples":[{"code_sha256_prefix":"72ab490c68e07338","entry":"reduce_fn","repo":"callmefish/NFnet","repo_kind":"listed","path":"utils.py","file_url":"https://github.com/callmefish/NFnet/blob/HEAD/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"72ab490c68e07338"}},{"code_sha256_prefix":"1f05ed2c304f566d","entry":"softmax_cross_entropy","repo":"callmefish/NFnet","repo_kind":"listed","path":"utils.py","file_url":"https://github.com/callmefish/NFnet/blob/HEAD/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"1f05ed2c304f566d"}},{"code_sha256_prefix":"07d4886e953a2f31","entry":"topk_correct","repo":"callmefish/NFnet","repo_kind":"listed","path":"utils.py","file_url":"https://github.com/callmefish/NFnet/blob/HEAD/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"07d4886e953a2f31"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}