Papers › Trainable Activations for Image Classification
Trainable Activations for Image Classification
Evgenii Pishchik
Non-linear activation functions are one of the main parts of deep neural network architectures. The choice of the activation function can affect model speed, performance and convergence. Most popular activation functions don't have any trainable parameters and don't alter during the training. We propose different activation functions with and without trainable parameters. Said activation functions have a number of advantages and disadvantages. We'll be testing the performance of said activation functions and comparing the results with widely known activation function ReLU. We assume that the activation functions with trainable parameters can outperform functions without ones, because the trainable parameters allow the model to "select'' the type of each of the activation functions itself, however, this strongly depends on the architecture of the deep neural network and the activation function itself.
Code
Repository list and official/mentioned flags are the archive's, frozen 2025-07-28. Reachability, where shown, is from one Syntology probe window (2026-09-16 to 2026-09-18); repositories not probed show nothing. GitHub stars are not tracked.
Code Syntology ran Syntology
Not run by Syntology. Nothing on this page verifies that the listed code works.
Tasks
Results from the paper archive 2025-07-28
| Task | Dataset | Model | Metric | Value | Rank at snapshot | Leaderboard | Report |
|---|---|---|---|---|---|---|---|
| Image Classification | CIFAR-10 | ResNet-26 (Trainable Activations) | Percentage correct | 91.1 | #196 of 265 | Archive leaderboard | report |
| Image Classification | CIFAR-10 | ResNet-32 (Trainable Activations) | Percentage correct | 90.9 | #197 of 265 | Archive leaderboard | report |
| Image Classification | CIFAR-10 | ResNet-44 (Trainable Activations) | Percentage correct | 90.5 | #204 of 265 | Archive leaderboard | report |
| Image Classification | CIFAR-10 | ResNet-20 (Trainable Activations) | Percentage correct | 90.4 | #205 of 265 | Archive leaderboard | report |
| Image Classification | CIFAR-10 | ResNet-14 (Trainable Activations) | Percentage correct | 89.0 | #212 of 265 | Archive leaderboard | report |
| Image Classification | CIFAR-10 | ResNet-56 (Trainable Activations) | Percentage correct | 88.8 | #215 of 265 | Archive leaderboard | report |
| Image Classification | CIFAR-10 | ResNet-8 (Trainable Activations) | Percentage correct | 86.5 | #228 of 265 | Archive leaderboard | report |
| Image Classification | MNIST | DNN-5 (Trainable Activations) | Accuracy | 97.2 | #59 of 81 | Archive leaderboard | report |
| Image Classification | MNIST | DNN-5 (Trainable Activations) | Percentage error | 2.8 | #59 of 81 | Archive leaderboard | report |
| Image Classification | MNIST | DNN-5 (Trainable Activations) | Trainable Parameters | 575051 | #59 of 81 | Archive leaderboard | report |
| Image Classification | MNIST | DNN-3 (Trainable Activations) | Accuracy | 97.0 | #60 of 81 | Archive leaderboard | report |
| Image Classification | MNIST | DNN-3 (Trainable Activations) | Percentage error | 3.0 | #60 of 81 | Archive leaderboard | report |
| Image Classification | MNIST | DNN-3 (Trainable Activations) | Trainable Parameters | 386719 | #60 of 81 | Archive leaderboard | report |
| Image Classification | MNIST | DNN-2 (Trainable Activations) | Accuracy | 96.4 | #61 of 81 | Archive leaderboard | report |
| Image Classification | MNIST | DNN-2 (Trainable Activations) | Percentage error | 3.6 | #61 of 81 | Archive leaderboard | report |
| Image Classification | MNIST | DNN-2 (Trainable Activations) | Trainable Parameters | 311651 | #61 of 81 | Archive leaderboard | report |
Ranks are positions in the archive's leaderboards as they stood at the 2025-07-28 snapshot. Results published since then are not among these rows, so a rank here is not a current standing.
Methods
Introduced by this paper: CosLU, DELU, LinComb, NormLinComb, ReLUN, ScaledSoftSign, ShiLU
Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections