{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/fast-and-accurate-deep-network-learning-by","title":"Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)","arxiv_id":"1511.07289","date":"2015-11-23","proceeding":null,"authors":["Djork-Arné Clevert","Thomas Unterthiner","Sepp Hochreiter"],"abstract":"We introduce the \"exponential linear unit\" (ELU) which speeds up learning in\ndeep neural networks and leads to higher classification accuracies. Like\nrectified linear units (ReLUs), leaky ReLUs (LReLUs) and parametrized ReLUs\n(PReLUs), ELUs alleviate the vanishing gradient problem via the identity for\npositive values. However, ELUs have improved learning characteristics compared\nto the units with other activation functions. In contrast to ReLUs, ELUs have\nnegative values which allows them to push mean unit activations closer to zero\nlike batch normalization but with lower computational complexity. Mean shifts\ntoward zero speed up learning by bringing the normal gradient closer to the\nunit natural gradient because of a reduced bias shift effect. While LReLUs and\nPReLUs have negative values, too, they do not ensure a noise-robust\ndeactivation state. ELUs saturate to a negative value with smaller inputs and\nthereby decrease the forward propagated variation and information. Therefore,\nELUs code the degree of presence of particular phenomena in the input, while\nthey do not quantitatively model the degree of their absence. In experiments,\nELUs lead not only to faster learning, but also to significantly better\ngeneralization performance than ReLUs and LReLUs on networks with more than 5\nlayers. On CIFAR-100 ELUs networks significantly outperform ReLU networks with\nbatch normalization while batch normalization does not improve ELU networks.\nELU networks are among the top 10 reported CIFAR-10 results and yield the best\npublished result on CIFAR-100, without resorting to multi-view evaluation or\nmodel averaging. On ImageNet, ELU networks considerably speed up learning\ncompared to a ReLU network with the same architecture, obtaining less than 10%\nclassification error for a single crop, single model network.","url_abs":"http://arxiv.org/abs/1511.07289v5","url_pdf":"http://arxiv.org/pdf/1511.07289v5.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"fast-and-accurate-deep-network-learning-by","repo_url":"https://github.com/AsafBarZvi/Liver_project","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"tf","reach":{"status":"ok"}},{"paper_slug":"fast-and-accurate-deep-network-learning-by","repo_url":"https://github.com/MalayAgarwal-Lee/MesoNet-DeepFakeDetection","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"tf","reach":null},{"paper_slug":"fast-and-accurate-deep-network-learning-by","repo_url":"https://github.com/MalayAgarwal-Lee/SteeringWheelAnglePredictor","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"tf","reach":null},{"paper_slug":"fast-and-accurate-deep-network-learning-by","repo_url":"https://github.com/andreblumenau/Contemporary-Neural-Network","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok"}},{"paper_slug":"fast-and-accurate-deep-network-learning-by","repo_url":"https://github.com/awur978/Autoencoder","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"tf","reach":{"status":"ok"}},{"paper_slug":"fast-and-accurate-deep-network-learning-by","repo_url":"https://github.com/ayushdewan/Latin-Alphabet-Recognition-Using-Convolutional-Neural-Networks-in-Tensorflow","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"tf","reach":{"status":"ok"}},{"paper_slug":"fast-and-accurate-deep-network-learning-by","repo_url":"https://github.com/barrykidney/CarND-Behavioral-Cloning-P3","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"fast-and-accurate-deep-network-learning-by","repo_url":"https://github.com/chuk-yong/ELU-vs-ReLU-for-Image-Recognition","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok"}},{"paper_slug":"fast-and-accurate-deep-network-learning-by","repo_url":"https://github.com/chuk-yong/keras-relu-elu-comparison","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok"}},{"paper_slug":"fast-and-accurate-deep-network-learning-by","repo_url":"https://github.com/hughperkins/DeepCL","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok","spdx":"MPL-2.0"}},{"paper_slug":"fast-and-accurate-deep-network-learning-by","repo_url":"https://github.com/ken-power/CVND-FacialKeypointDetection","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}},{"paper_slug":"fast-and-accurate-deep-network-learning-by","repo_url":"https://github.com/sinking8/Lenet5","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok"}},{"paper_slug":"fast-and-accurate-deep-network-learning-by","repo_url":"https://github.com/stefangeneralao/exterior-interior-classifier","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok"}},{"paper_slug":"fast-and-accurate-deep-network-learning-by","repo_url":"https://github.com/vamsiramakrishnan/BehavioralCloning","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"tf","reach":{"status":"ok"}},{"paper_slug":"fast-and-accurate-deep-network-learning-by","repo_url":"https://github.com/xiexiexiaoxiexie/Udacity-self-driving-car-engineer-P4-Behavioral-Cloning","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok"}},{"paper_slug":"fast-and-accurate-deep-network-learning-by","repo_url":"https://github.com/MaximeVandegar/Papers-in-100-Lines-of-Code/tree/main/Fast_and_Accurate_Deep_Network_Learning_by_Exponential_Linear_Units_ELUs","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"classification","task_name":"General Classification"},{"task_slug":"image-classification","task_name":"Image Classification"}],"methods":[{"method_slug":"batch-normalization","method_name":"Batch Normalization"},{"method_slug":"elu","method_name":"ELU"},{"method_slug":"relu","method_name":"ReLU"},{"method_slug":"speed","method_name":"SPEED"}],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/image-classification-on-cifar-10","task":"Image Classification","dataset":"CIFAR-10","model":"Exponential Linear Units","rank_in_archive_order":174,"of":265,"metrics":{"Percentage correct":"93.5"},"uses_additional_data":false},{"leaderboard":"/sota/image-classification-on-cifar-100","task":"Image Classification","dataset":"CIFAR-100","model":"Exponential Linear Units","rank_in_archive_order":151,"of":211,"metrics":{"Percentage correct":"75.7"},"uses_additional_data":true}],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=1511.07289","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"1511.07289"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/awur978/Autoencoder","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/ken-power/CVND-FacialKeypointDetection","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/barrykidney/CarND-Behavioral-Cloning-P3","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/ayushdewan/Latin-Alphabet-Recognition-Using-Convolutional-Neural-Networks-in-Tensorflow","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/xiexiexiaoxiexie/Udacity-self-driving-car-engineer-P4-Behavioral-Cloning","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/MalayAgarwal-Lee/SteeringWheelAnglePredictor","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/chuk-yong/ELU-vs-ReLU-for-Image-Recognition","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/MaximeVandegar/Papers-in-100-Lines-of-Code/tree/main/Fast_and_Accurate_Deep_Network_Learning_by_Exponential_Linear_Units_ELUs","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/chuk-yong/keras-relu-elu-comparison","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/vamsiramakrishnan/BehavioralCloning","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/hughperkins/DeepCL","reach":{"status":"ok","spdx":"MPL-2.0"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/stefangeneralao/exterior-interior-classifier","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/andreblumenau/Contemporary-Neural-Network","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/AsafBarZvi/Liver_project","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/MalayAgarwal-Lee/MesoNet-DeepFakeDetection","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/sinking8/Lenet5","reach":{"status":"ok"}}],"summary":{"ran":1,"ran_draft_wrong":2,"unverified":4},"by_repo_kind":{"listed":{"samples":7,"ran":3,"repositories":4}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":2,"samples":[{"code_sha256_prefix":"ecd258475c1c5865","entry":"ELU","repo":"MaximeVandegar/Papers-in-100-Lines-of-Code","repo_kind":"listed","path":"Fast_and_Accurate_Deep_Network_Learning_by_Exponential_Linear_Units_ELUs/elu.py","file_url":"https://github.com/MaximeVandegar/Papers-in-100-Lines-of-Code/blob/HEAD/Fast_and_Accurate_Deep_Network_Learning_by_Exponential_Linear_Units_ELUs/elu.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ecd258475c1c5865"}},{"code_sha256_prefix":"5770a3f04d128ada","entry":"activation_layer","repo":"MalayAgarwal-Lee/MesoNet-DeepFakeDetection","repo_kind":"listed","path":"mesonet/model.py","file_url":"https://github.com/MalayAgarwal-Lee/MesoNet-DeepFakeDetection/blob/HEAD/mesonet/model.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"5770a3f04d128ada"}},{"code_sha256_prefix":"153c0f814833ed63","entry":"activation_layer","repo":"MalayAgarwal-Lee/SteeringWheelAnglePredictor","repo_kind":"listed","path":"model.py","file_url":"https://github.com/MalayAgarwal-Lee/SteeringWheelAnglePredictor/blob/HEAD/model.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"153c0f814833ed63"}},{"code_sha256_prefix":"7ceb60b96974b32c","entry":"load_data","repo":"barrykidney/CarND-Behavioral-Cloning-P3","repo_kind":"listed","path":"model.py","file_url":"https://github.com/barrykidney/CarND-Behavioral-Cloning-P3/blob/HEAD/model.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"7ceb60b96974b32c"}},{"code_sha256_prefix":"2a1afa2d5483afe7","entry":"load_driving_log_files","repo":"barrykidney/CarND-Behavioral-Cloning-P3","repo_kind":"listed","path":"model.py","file_url":"https://github.com/barrykidney/CarND-Behavioral-Cloning-P3/blob/HEAD/model.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"2a1afa2d5483afe7"}},{"code_sha256_prefix":"7f3ca20b354d2c4b","entry":"process_image","repo":"barrykidney/CarND-Behavioral-Cloning-P3","repo_kind":"listed","path":"model.py","file_url":"https://github.com/barrykidney/CarND-Behavioral-Cloning-P3/blob/HEAD/model.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"7f3ca20b354d2c4b"}},{"code_sha256_prefix":"8cb7962d35c1053c","entry":"process_image","repo":"barrykidney/CarND-Behavioral-Cloning-P3","repo_kind":"listed","path":"drive.py","file_url":"https://github.com/barrykidney/CarND-Behavioral-Cloning-P3/blob/HEAD/drive.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"8cb7962d35c1053c"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}