{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/empirical-evaluation-of-rectified-activations","title":"Empirical Evaluation of Rectified Activations in Convolutional Network","arxiv_id":"1505.00853","date":"2015-05-05","proceeding":null,"authors":["Bing Xu","Naiyan Wang","Tianqi Chen","Mu Li"],"abstract":"In this paper we investigate the performance of different types of rectified\nactivation functions in convolutional neural network: standard rectified linear\nunit (ReLU), leaky rectified linear unit (Leaky ReLU), parametric rectified\nlinear unit (PReLU) and a new randomized leaky rectified linear units (RReLU).\nWe evaluate these activation function on standard image classification task.\nOur experiments suggest that incorporating a non-zero slope for negative part\nin rectified activation units could consistently improve the results. Thus our\nfindings are negative on the common belief that sparsity is the key of good\nperformance in ReLU. Moreover, on small scale dataset, using deterministic\nnegative slope or learning it are both prone to overfitting. They are not as\neffective as using their randomized counterpart. By using RReLU, we achieved\n75.68\\% accuracy on CIFAR-100 test set without multiple test or ensemble.","url_abs":"http://arxiv.org/abs/1505.00853v2","url_pdf":"http://arxiv.org/pdf/1505.00853v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"empirical-evaluation-of-rectified-activations","repo_url":"https://github.com/OsvaldN/APS360_Project","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"empirical-evaluation-of-rectified-activations","repo_url":"https://github.com/spinterRu/fashion_mnist","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"tf","reach":{"status":"ok"}}],"tasks":[{"task_slug":"classification","task_name":"General Classification"},{"task_slug":"image-classification","task_name":"Image Classification"},{"task_slug":"image-classification","task_name":"image-classification"}],"methods":[{"method_slug":"rrelu","method_name":"RReLU"},{"method_slug":"relu","method_name":"ReLU"}],"datasets_introduced":[],"methods_introduced":[{"slug":"rrelu","name":"RReLU","full_name":"Randomized Leaky Rectified Linear Units"}],"results":[{"leaderboard":"/sota/image-classification-on-cifar-10","task":"Image Classification","dataset":"CIFAR-10","model":"RReLU","rank_in_archive_order":214,"of":265,"metrics":{"Percentage correct":"88.8"},"uses_additional_data":false},{"leaderboard":"/sota/image-classification-on-cifar-100","task":"Image Classification","dataset":"CIFAR-100","model":"RReLU","rank_in_archive_order":201,"of":211,"metrics":{"Percentage correct":"59.8"},"uses_additional_data":false}],"syntology":{"syntology_url":"https://syntology.ai/paper/1505.00853","atlas_url":"https://app.syntology.ai/?focus=1505.00853","mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}