{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/pade-activation-units-end-to-end-learning-of","title":"Padé Activation Units: End-to-end Learning of Flexible Activation Functions in Deep Networks","arxiv_id":"1907.06732","date":"2019-07-15","proceeding":"ICLR 2020 1","authors":["Alejandro Molina","Patrick Schramowski","Kristian Kersting"],"abstract":"The performance of deep network learning strongly depends on the choice of the non-linear activation function associated with each neuron. However, deciding on the best activation is non-trivial, and the choice depends on the architecture, hyper-parameters, and even on the dataset. Typically these activations are fixed by hand before training. Here, we demonstrate how to eliminate the reliance on first picking fixed activation functions by using flexible parametric rational functions instead. The resulting Pad\\'e Activation Units (PAUs) can both approximate common activation functions and also learn new ones while providing compact representations. Our empirical evidence shows that end-to-end learning deep networks with PAUs can increase the predictive performance. Moreover, PAUs pave the way to approximations with provable robustness. https://github.com/ml-research/pau","url_abs":"https://arxiv.org/abs/1907.06732v3","url_pdf":"https://arxiv.org/pdf/1907.06732v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"pade-activation-units-end-to-end-learning-of","repo_url":"https://github.com/ml-research/pau","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}},{"paper_slug":"pade-activation-units-end-to-end-learning-of","repo_url":"https://github.com/ml-research/rational_activations","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"pade-activation-units-end-to-end-learning-of","repo_url":"https://github.com/ml-research/rational_sl","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"unanswered"}},{"paper_slug":"pade-activation-units-end-to-end-learning-of","repo_url":"https://github.com/ChristophReich1996/Pade-Activation-Unit","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"pytorch","reach":null},{"paper_slug":"pade-activation-units-end-to-end-learning-of","repo_url":"https://github.com/ChristophReich1996/ToeffiPy/blob/master/autograd/nn/activation.py","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[],"methods":[{"method_slug":"pau","method_name":"PAU"},{"method_slug":"rational-activation-functions","method_name":"Rational Activation Function"},{"method_slug":"rational-activation-functions","method_name":"Rational Activation function"}],"datasets_introduced":[],"methods_introduced":[{"slug":"pau","name":"PAU","full_name":"Padé Activation Units"},{"slug":"rational-activation-functions","name":"Rational Activation function","full_name":"Rational Activation function"}],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/1907.06732","atlas_url":"https://app.syntology.ai/?focus=1907.06732","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"1907.06732"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/ml-research/rational_activations","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/ChristophReich1996/ToeffiPy/blob/master/autograd/nn/activation.py","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/ChristophReich1996/Pade-Activation-Unit","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/ml-research/pau","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/ml-research/rational_sl","reach":{"status":"unanswered"}}],"summary":{"ran_fixture":3,"unverified":1},"by_repo_kind":{"listed":{"samples":4,"ran":3,"repositories":2}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"39ec3d1f9d97cc5c","entry":"Rational_PYTORCH_A_F","repo":"ml-research/rational_activations","repo_kind":"listed","path":"rational/torch/rational_pytorch_functions.py","file_url":"https://github.com/ml-research/rational_activations/blob/HEAD/rational/torch/rational_pytorch_functions.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"39ec3d1f9d97cc5c"}},{"code_sha256_prefix":"15a510212432a6c3","entry":"Rational_PYTORCH_B_F","repo":"ml-research/rational_activations","repo_kind":"listed","path":"rational/torch/rational_pytorch_functions.py","file_url":"https://github.com/ml-research/rational_activations/blob/HEAD/rational/torch/rational_pytorch_functions.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"15a510212432a6c3"}},{"code_sha256_prefix":"6e3e9a40d8b5f9f7","entry":"Rational_PYTORCH_C_F","repo":"ml-research/rational_activations","repo_kind":"listed","path":"rational/torch/rational_pytorch_functions.py","file_url":"https://github.com/ml-research/rational_activations/blob/HEAD/rational/torch/rational_pytorch_functions.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"6e3e9a40d8b5f9f7"}},{"code_sha256_prefix":"e09004320aabd5a6","entry":"freeze_pau","repo":"ChristophReich1996/Pade-Activation-Unit","repo_kind":"listed","path":"pau/pau.py","file_url":"https://github.com/ChristophReich1996/Pade-Activation-Unit/blob/HEAD/pau/pau.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e09004320aabd5a6"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}