Papers › Automatic Differentiation for Tensor Algebras
Automatic Differentiation for Tensor Algebras
Sebastian Urban, Patrick van der Smagt
Kjolstad et. al. proposed a tensor algebra compiler. It takes expressions that define a tensor element-wise, such as fᵢⱼ(a,b,c,d) = exp[-∑ₖ₌₀⁴ ((aᵢₖ+bⱼₖ)² cᵢᵢ + dᵢ₊ₖ³ ) ], and generates the corresponding compute kernel code. For machine learning, especially deep learning, it is often necessary to compute the gradient of a loss function l(a,b,c,d)=l(f(a,b,c,d)) with respect to parameters a,b,c,d. If tensor compilers are to be applied in this field, it is necessary to derive expressions for the derivatives of element-wise defined tensors, i.e. expressions for (da)ᵢₖ=∂l/∂aᵢₖ. When the mapping between function indices and argument indices is not 1:1, special attention is required. For the function fᵢⱼ (x) = xᵢ², the derivative of the loss is (dx)ᵢ=∂l/∂xᵢ=∑ⱼ (df)ᵢⱼ2xᵢ; the sum is necessary because index j does not appear in the indices of f. Another example is fᵢ(x)=xᵢᵢ², where x is a matrix; here we have (dx)ᵢⱼ=δᵢⱼ(df)ᵢ2xᵢᵢ; the Kronecker delta is necessary because the derivative is zero for off-diagonal elements. Another indexing scheme is used by fᵢⱼ(x)=expxᵢ₊ⱼ; here the correct derivative is (dx)ₖ=∑ᵢ (df)_(i,k-i) expxₖ, where the range of the sum must be chosen appropriately. In this publication we present an algorithm that can handle any case in which the indices of an argument are an arbitrary linear combination of the indices of the function, thus all the above examples can be handled. Sums (and their ranges) and Kronecker deltas are automatically inserted into the derivatives as necessary. Additionally, the indices are transformed, if required (as in the last example). The algorithm outputs a symbolic expression that can be subsequently fed into a tensor algebra compiler. Source code is provided.
Code
18 repositories listed; official and paper-mentioned ones first.
Repository list and official/mentioned flags are the archive's, frozen 2025-07-28. Reachability, where shown, is from one Syntology probe window (2026-09-16 to 2026-09-18); repositories not probed show nothing. GitHub stars are not tracked.
Code Syntology ran Syntology
Not run by Syntology. Nothing on this page verifies that the listed code works.
Tasks
Results from the paper archive 2025-07-28
No leaderboard rows for this paper in the archive.
Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections