{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/smooth-activations-and-reproducibility-in-1","title":"Smooth activations and reproducibility in deep networks","arxiv_id":"2010.09931","date":"2020-10-20","proceeding":null,"authors":["Gil I. Shamir","Dong Lin","Lorenzo Coviello"],"abstract":"Deep networks are gradually penetrating almost every domain in our lives due to their amazing success. However, with substantive performance accuracy improvements comes the price of \\emph{irreproducibility}. Two identical models, trained on the exact same training dataset may exhibit large differences in predictions on individual examples even when average accuracy is similar, especially when trained on highly distributed parallel systems. The popular Rectified Linear Unit (ReLU) activation has been key to recent success of deep networks. We demonstrate, however, that ReLU is also a catalyzer to irreproducibility in deep networks. We show that not only can activations smoother than ReLU provide better accuracy, but they can also provide better accuracy-reproducibility tradeoffs. We propose a new family of activations; Smooth ReLU (\\emph{SmeLU}), designed to give such better tradeoffs, while also keeping the mathematical expression simple, and thus implementation cheap. SmeLU is monotonic, mimics ReLU, while providing continuous gradients, yielding better reproducibility. We generalize SmeLU to give even more flexibility and then demonstrate that SmeLU and its generalized form are special cases of a more general methodology of REctified Smooth Continuous Unit (RESCU) activations. Empirical results demonstrate the superior accuracy-reproducibility tradeoffs with smooth activations, SmeLU in particular.","url_abs":"https://arxiv.org/abs/2010.09931v2","url_pdf":"https://arxiv.org/pdf/2010.09931v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"smooth-activations-and-reproducibility-in-1","repo_url":"https://github.com/MindSpore-scientific/code-13/tree/main/Super-Resolution-for-Root-Imaging","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"mindspore","reach":null},{"paper_slug":"smooth-activations-and-reproducibility-in-1","repo_url":"https://github.com/MindSpore-scientific/code-8/tree/main/SmeLU-master","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"mindspore","reach":null},{"paper_slug":"smooth-activations-and-reproducibility-in-1","repo_url":"https://github.com/awsaf49/tensorflow_extra","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"tf","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"smooth-activations-and-reproducibility-in-1","repo_url":"https://github.com/pwc-1/Paper-9/tree/main/7/SmeLU-master","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"mindspore","reach":null}],"tasks":[],"methods":[{"method_slug":"relu","method_name":"ReLU"},{"method_slug":"smelu","method_name":"SmeLU"},{"method_slug":"tanh-activation","method_name":"Tanh Activation"},{"method_slug":"tanhexp","method_name":"TanhExp"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2010.09931","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2010.09931"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/MindSpore-scientific/code-13/tree/main/Super-Resolution-for-Root-Imaging","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/awsaf49/tensorflow_extra","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/pwc-1/Paper-9/tree/main/7/SmeLU-master","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/MindSpore-scientific/code-8/tree/main/SmeLU-master","reach":null}],"summary":{"unverified":3},"by_repo_kind":{"listed":{"samples":3,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"a6b552197a11122f","entry":"random_float","repo":"awsaf49/tensorflow_extra","repo_kind":"listed","path":"tensorflow_extra/utils.py","file_url":"https://github.com/awsaf49/tensorflow_extra/blob/HEAD/tensorflow_extra/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a6b552197a11122f"}},{"code_sha256_prefix":"26e4ca653127aa07","entry":"random_int","repo":"awsaf49/tensorflow_extra","repo_kind":"listed","path":"tensorflow_extra/utils.py","file_url":"https://github.com/awsaf49/tensorflow_extra/blob/HEAD/tensorflow_extra/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"26e4ca653127aa07"}},{"code_sha256_prefix":"11eef4f59aa09dd7","entry":"smelu","repo":"awsaf49/tensorflow_extra","repo_kind":"listed","path":"tensorflow_extra/activations.py","file_url":"https://github.com/awsaf49/tensorflow_extra/blob/HEAD/tensorflow_extra/activations.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"11eef4f59aa09dd7"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}