{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/adaptive-parametric-activation","title":"Adaptive Parametric Activation","arxiv_id":"2407.08567","date":"2024-07-11","proceeding":null,"authors":["Konstantinos Panagiotis Alexandridis","Jiankang Deng","Anh Nguyen","Shan Luo"],"abstract":"The activation function plays a crucial role in model optimisation, yet the optimal choice remains unclear. For example, the Sigmoid activation is the de-facto activation in balanced classification tasks, however, in imbalanced classification, it proves inappropriate due to bias towards frequent classes. In this work, we delve deeper in this phenomenon by performing a comprehensive statistical analysis in the classification and intermediate layers of both balanced and imbalanced networks and we empirically show that aligning the activation function with the data distribution, enhances the performance in both balanced and imbalanced tasks. To this end, we propose the Adaptive Parametric Activation (APA) function, a novel and versatile activation function that unifies most common activation functions under a single formula. APA can be applied in both intermediate layers and attention layers, significantly outperforming the state-of-the-art on several imbalanced benchmarks such as ImageNet-LT, iNaturalist2018, Places-LT, CIFAR100-LT and LVIS and balanced benchmarks such as ImageNet1K, COCO and V3DET. The code is available at https://github.com/kostas1515/AGLU.","url_abs":"https://arxiv.org/abs/2407.08567v2","url_pdf":"https://arxiv.org/pdf/2407.08567v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"adaptive-parametric-activation","repo_url":"https://github.com/kostas1515/aglu","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"instance-segmentation","task_name":"Instance Segmentation"},{"task_slug":"long-tail-learning","task_name":"Long-tail Learning"},{"task_slug":"imbalanced-classification","task_name":"imbalanced classification"}],"methods":[{"method_slug":"aglu","method_name":"AGLU"},{"method_slug":"apa","method_name":"APA"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"sigmoid-activation","method_name":"Sigmoid Activation"},{"method_slug":"softmax","method_name":"Softmax"}],"datasets_introduced":[],"methods_introduced":[{"slug":"aglu","name":"AGLU","full_name":"Adaptive Generalised Linear Unit"}],"results":[{"leaderboard":"/sota/instance-segmentation-on-lvis-v1-0-val","task":"Instance Segmentation","dataset":"LVIS v1.0 val","model":"SE-R101-FPN-MaskRCNN-APA","rank_in_archive_order":15,"of":25,"metrics":{"mask AP":"30.7","mask APr":"23.6"},"uses_additional_data":false},{"leaderboard":"/sota/instance-segmentation-on-lvis-v1-0-val","task":"Instance Segmentation","dataset":"LVIS v1.0 val","model":"SE-R50-FPN-MaskRCNN-APA","rank_in_archive_order":16,"of":25,"metrics":{"mask AP":"29.1","mask APr":"21.6"},"uses_additional_data":false},{"leaderboard":"/sota/long-tail-learning-on-imagenet-lt","task":"Long-tail Learning","dataset":"ImageNet-LT","model":"APA (SE-ResNext-50)","rank_in_archive_order":17,"of":69,"metrics":{"Top-1 Accuracy":"59.1"},"uses_additional_data":false},{"leaderboard":"/sota/long-tail-learning-on-imagenet-lt","task":"Long-tail Learning","dataset":"ImageNet-LT","model":"APA (SE-ResNet-50)","rank_in_archive_order":25,"of":69,"metrics":{"Top-1 Accuracy":"57.9"},"uses_additional_data":false},{"leaderboard":"/sota/long-tail-learning-on-places-lt","task":"Long-tail Learning","dataset":"Places-LT","model":"APA (SE-ResNet-50)","rank_in_archive_order":11,"of":29,"metrics":{"Top-1 Accuracy":"42.0"},"uses_additional_data":false},{"leaderboard":"/sota/long-tail-learning-on-inaturalist-2018","task":"Long-tail Learning","dataset":"iNaturalist 2018","model":"APA (SE-ResNet-50)","rank_in_archive_order":16,"of":43,"metrics":{"Top-1 Accuracy":"74.8"},"uses_additional_data":false}],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2407.08567","mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}