{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/trainable-highly-expressive-activation","title":"Trainable Highly-expressive Activation Functions","arxiv_id":"2407.07564","date":"2024-07-10","proceeding":null,"authors":["Irit Chelly","Shahaf E. Finder","Shira Ifergane","Oren Freifeld"],"abstract":"Nonlinear activation functions are pivotal to the success of deep neural nets, and choosing the appropriate activation function can significantly affect their performance. Most networks use fixed activation functions (e.g., ReLU, GELU, etc.), and this choice might limit their expressiveness. Furthermore, different layers may benefit from diverse activation functions. Consequently, there has been a growing interest in trainable activation functions. In this paper, we introduce DiTAC, a trainable highly-expressive activation function based on an efficient diffeomorphic transformation (called CPAB). Despite introducing only a negligible number of trainable parameters, DiTAC enhances model expressiveness and performance, often yielding substantial improvements. It also outperforms existing activation functions (regardless whether the latter are fixed or trainable) in tasks such as semantic segmentation, image generation, regression problems, and image classification. Our code is available at https://github.com/BGU-CS-VIL/DiTAC.","url_abs":"https://arxiv.org/abs/2407.07564v2","url_pdf":"https://arxiv.org/pdf/2407.07564v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"trainable-highly-expressive-activation","repo_url":"https://github.com/bgu-cs-vil/ditac","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"image-classification","task_name":"Image Classification"},{"task_slug":"image-generation","task_name":"Image Generation"},{"task_slug":"semantic-segmentation","task_name":"Semantic Segmentation"},{"task_slug":"image-classification","task_name":"image-classification"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2407.07564","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2407.07564"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/bgu-cs-vil/ditac","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":8},"by_repo_kind":{"official":{"samples":8,"ran":8,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"add0d777b27172fa","entry":"R2","repo":"bgu-cs-vil/ditac","repo_kind":"official","path":"regression_example/evaluation.py","file_url":"https://github.com/bgu-cs-vil/ditac/blob/HEAD/regression_example/evaluation.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"add0d777b27172fa"}},{"code_sha256_prefix":"0088fd84030dee57","entry":"RMSE","repo":"bgu-cs-vil/ditac","repo_kind":"official","path":"regression_example/evaluation.py","file_url":"https://github.com/bgu-cs-vil/ditac/blob/HEAD/regression_example/evaluation.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"0088fd84030dee57"}},{"code_sha256_prefix":"442198b32b497424","entry":"compute_loss","repo":"bgu-cs-vil/ditac","repo_kind":"official","path":"regression_example/evaluation.py","file_url":"https://github.com/bgu-cs-vil/ditac/blob/HEAD/regression_example/evaluation.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"442198b32b497424"}},{"code_sha256_prefix":"97af7e6c82e34298","entry":"get_fast_transform_grid_eval","repo":"bgu-cs-vil/ditac","repo_kind":"official","path":"ditac/utils/fast_cpab.py","file_url":"https://github.com/bgu-cs-vil/ditac/blob/HEAD/ditac/utils/fast_cpab.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"97af7e6c82e34298"}},{"code_sha256_prefix":"1d87ab654455f87b","entry":"get_precision_mat","repo":"bgu-cs-vil/ditac","repo_kind":"official","path":"ditac/utils/misc.py","file_url":"https://github.com/bgu-cs-vil/ditac/blob/HEAD/ditac/utils/misc.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"1d87ab654455f87b"}},{"code_sha256_prefix":"3b035cc02d994e08","entry":"norm_data","repo":"bgu-cs-vil/ditac","repo_kind":"official","path":"ditac/utils/misc.py","file_url":"https://github.com/bgu-cs-vil/ditac/blob/HEAD/ditac/utils/misc.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"3b035cc02d994e08"}},{"code_sha256_prefix":"57b024a42d9d2520","entry":"phi","repo":"bgu-cs-vil/ditac","repo_kind":"official","path":"ditac/utils/misc.py","file_url":"https://github.com/bgu-cs-vil/ditac/blob/HEAD/ditac/utils/misc.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"57b024a42d9d2520"}},{"code_sha256_prefix":"c40a4a89747caf9e","entry":"to","repo":"bgu-cs-vil/ditac","repo_kind":"official","path":"ditac/utils/cpab_t.py","file_url":"https://github.com/bgu-cs-vil/ditac/blob/HEAD/ditac/utils/cpab_t.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c40a4a89747caf9e"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}