{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/design-space-exploration-of-neural-network","title":"Design Space Exploration of Neural Network Activation Function Circuits","arxiv_id":"1810.08650","date":"2018-09-22","proceeding":null,"authors":["Yang Tao","Wei Yadong","Tu Zhijun","Zeng Haolun","Kinsy Michel A.","Zheng Nanning","Ren Pengju"],"abstract":"The widespread application of artificial neural networks has prompted\nresearchers to experiment with FPGA and customized ASIC designs to speed up\ntheir computation. These implementation efforts have generally focused on\nweight multiplication and signal summation operations, and less on activation\nfunctions used in these applications. Yet, efficient hardware implementations\nof nonlinear activation functions like Exponential Linear Units (ELU), Scaled\nExponential Linear Units (SELU), and Hyperbolic Tangent (tanh), are central to\ndesigning effective neural network accelerators, since these functions require\nlots of resources. In this paper, we explore efficient hardware implementations\nof activation functions using purely combinational circuits, with a focus on\ntwo widely used nonlinear activation functions, i.e., SELU and tanh. Our\nexperiments demonstrate that neural networks are generally insensitive to the\nprecision of the activation function. The results also prove that the proposed\ncombinational circuit-based approach is very efficient in terms of speed and\narea, with negligible accuracy loss on the MNIST, CIFAR-10 and IMAGENET\nbenchmarks. Synopsys Design Compiler synthesis results show that circuit\ndesigns for tanh and SELU can save between 3.13-7.69 and 4.45-8:45 area\ncompared to the LUT/memory-based implementations, and can operate at 5.14GHz\nand 4.52GHz using the 28nm SVT library, respectively. The implementation is\navailable at: https://github.com/ThomasMrY/ActivationFunctionDemo.","url_abs":"http://arxiv.org/abs/1810.08650v1","url_pdf":"http://arxiv.org/pdf/1810.08650v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"design-space-exploration-of-neural-network","repo_url":"https://github.com/ThomasMrY/ActivationFunctionDemo","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":null}],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":null,"atlas_url":null,"mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}