Methods › General › Stochastic Optimization › Powerpropagation
Powerpropagation
Introduced by Jonathan Schwarz et al. in Powerpropagation: A sparsity inducing weight reparameterisation
archive 2025-07-28 Description, source and code snippet are the archive's method entry.
Powerpropagation is a weight-parameterisation for neural networks that leads to inherently sparse models. Exploiting the behaviour of gradient descent, it gives rise to weight updates exhibiting a “rich get richer” dynamic, leaving low-magnitude parameters largely unaffected by learning.In other words, parameters with larger magnitudes are allowed to adapt faster in order to represent the required features to solve the task, while smaller magnitude parameters are restricted, making it more likely that they will be irrelevant in representing the learned solution. Models trained in this manner exhibit similar performance, but have a distribution with markedly higher density at zero, allowing more parameters to be pruned safely.
Papers archive 2025-07-28
1 shown of 1, newest first. Repository counts are the archive's code-links table. A Syntology line states what Syntology ran from that paper's harvested code; it is per sample and not a correctness claim.
-
Powerpropagation: A sparsity inducing weight reparameterisation 1 Oct 2021 · 2 repositories · arXiv:2110.00296Syntology ran 1 of 1 samples · 0 unverified · 1 pointer-only (licence)
Tasks archive 2025-07-28
The archive attaches no task to a paper tagged with this method.
Usage over time archive 2025-07-28
Components: the archive holds no method-to-method composition, so PwC's Components table cannot be rebuilt; the Papers list carries no Results column for the same reason (the archive does not join its leaderboard rows to method tags).
Categories archive 2025-07-28
Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections