{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/frelu-flexible-rectified-linear-units-for","title":"FReLU: Flexible Rectified Linear Units for Improving Convolutional Neural Networks","arxiv_id":"1706.08098","date":"2017-06-25","proceeding":null,"authors":["Suo Qiu","Xiangmin Xu","Bolun Cai"],"abstract":"Rectified linear unit (ReLU) is a widely used activation function for deep\nconvolutional neural networks. However, because of the zero-hard rectification,\nReLU networks miss the benefits from negative values. In this paper, we propose\na novel activation function called \\emph{flexible rectified linear unit\n(FReLU)} to further explore the effects of negative values. By redesigning the\nrectified point of ReLU as a learnable parameter, FReLU expands the states of\nthe activation output. When the network is successfully trained, FReLU tends to\nconverge to a negative value, which improves the expressiveness and thus the\nperformance. Furthermore, FReLU is designed to be simple and effective without\nexponential functions to maintain low cost computation. For being able to\neasily used in various network architectures, FReLU does not rely on strict\nassumptions by self-adaption. We evaluate FReLU on three standard image\nclassification datasets, including CIFAR-10, CIFAR-100, and ImageNet.\nExperimental results show that the proposed method achieves fast convergence\nand higher performances on both plain and residual networks.","url_abs":"http://arxiv.org/abs/1706.08098v2","url_pdf":"http://arxiv.org/pdf/1706.08098v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"frelu-flexible-rectified-linear-units-for","repo_url":"https://github.com/ducha-aiki/caffenet-benchmark","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"none","reach":null},{"paper_slug":"frelu-flexible-rectified-linear-units-for","repo_url":"https://github.com/danFromTelAviv/FReLU-Keras","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"tf","reach":null}],"tasks":[{"task_slug":"image-classification","task_name":"Image Classification"},{"task_slug":"image-classification","task_name":"image-classification"}],"methods":[{"method_slug":"relu","method_name":"ReLU"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/1706.08098","atlas_url":"https://app.syntology.ai/?focus=1706.08098","mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}