Methods › General › Stochastic Optimization › Forward gradient
Forward gradient
Introduced by Atılım Güneş Baydin et al. in Gradients without Backpropagation
archive 2025-07-28 Description, source and code snippet are the archive's method entry.
Forward gradients are unbiased estimators of the gradient ∇f(θ) for a function f: ℝⁿ →ℝ, given by g(θ) = ⟨∇f(θ) , v ⟩v.
Here v = (v₁, …, vₙ) is a random vector, which must satisfy the following conditions in order for g(θ) to be an unbiased estimator of ∇f(θ)
- vᵢ ⊥vⱼ for all i ≠j
- 𝔼[vᵢ] = 0 for all i
- 𝕍[vᵢ] = 1 for all i
Forward gradients can be computed with a single jvp (Jacobian Vector Product), which enables the use of the forward mode of autodifferentiation instead of the usual reverse mode, which has worse computational characteristics.
Papers archive 2025-07-28
12 shown of 12, newest first. Repository counts are the archive's code-links table. A Syntology line states what Syntology ran from that paper's harvested code; it is per sample and not a correctness claim.
-
A Scalable Hybrid Training Approach for Recurrent Spiking Neural Networks 17 Jun 2025 · 1 repository · arXiv:2506.14464
-
Flexible and Efficient Surrogate Gradient Modeling with Forward Gradient Injection 31 May 2024 · 0 repositories · arXiv:2406.00177
-
Projected Forward Gradient-Guided Frank-Wolfe Algorithm via Variance Reduction 19 Mar 2024 · 0 repositories · arXiv:2403.12511
-
Revisiting Zeroth-Order Optimization for Memory-Efficient LLM Fine-Tuning: A Benchmark 18 Feb 2024 · 1 repository · arXiv:2402.11592Syntology ran 8 of 13 samples · 5 unverified · 13 pointer-only (licence)
-
ODICE: Revealing the Mystery of Distribution Correction Estimation via Orthogonal-gradient Update 1 Feb 2024 · 1 repository · arXiv:2402.00348Syntology ran 8 of 8 samples · 0 unverified · 8 pointer-only (licence)
-
Convergence guarantees for forward gradient descent in the linear regression model 26 Sep 2023 · 0 repositories · arXiv:2309.15001
-
Accelerated On-Device Forward Neural Network Training with Module-Wise Descending Asynchronism 21 Sep 2023 · 0 repositories
-
Can Forward Gradient Match Backpropagation? 12 Jun 2023 · 1 repository · arXiv:2306.06968
-
Low-Variance Forward Gradients using Direct Feedback Alignment and Momentum 14 Dec 2022 · 1 repository · arXiv:2212.07282
-
Scaling Forward Gradient With Local Losses 7 Oct 2022 · 1 repository · arXiv:2210.03310
-
Optimization without Backpropagation 13 Sep 2022 · 1 repository · arXiv:2209.06302Syntology ran 0 of 8 samples · 8 unverified
-
Gradients without Backpropagation 17 Feb 2022 · 3 repositories · arXiv:2202.08587Syntology ran 2 of 4 samples · 2 unverified
Tasks archive 2025-07-28
7 tasks the archive attaches to papers tagged with this method, by distinct papers. A task without a page in the catalog is plain text.
| Task | Papers |
|---|---|
| Benchmarking | 1 |
| Imitation Learning | 1 |
| Memorization | 1 |
| Model Optimization | 1 |
| Offline RL | 1 |
| Reinforcement Learning (RL) | 1 |
| regression | 1 |
Usage over time archive 2025-07-28
Components: the archive holds no method-to-method composition, so PwC's Components table cannot be rebuilt; the Papers list carries no Results column for the same reason (the archive does not join its leaderboard rows to method tags).
Categories archive 2025-07-28
Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections