Methods › General › Attention Mechanisms › Cross-Covariance Attention
Cross-Covariance Attention
archive 2025-07-28 Description, source and code snippet are the archive's method entry.
Cross-Covariance Attention, or XCA, is an attention mechanism which operates along the feature dimension instead of the token dimension as in conventional transformers.
Using the definitions of queries, keys and values from conventional attention, the cross-covariance attention function is defined as:
XC-Attention (Q, K, V)=V 𝒜_(XC)(K, Q), 𝒜_(XC)(K, Q)=Softmax(K̂^⊤ Q̂ / τ)
where each output token embedding is a convex combination of the dᵥ features of its corresponding token embedding in V. The attention weights 𝒜 are computed based on the cross-covariance matrix.
Papers archive 2025-07-28
7 shown of 7, newest first. Repository counts are the archive's code-links table. A Syntology line states what Syntology ran from that paper's harvested code; it is per sample and not a correctness claim.
-
Advancing Deformable Medical Image Registration with Multi-axis Cross-covariance Attention 24 Dec 2024 · 1 repository · arXiv:2412.18545
-
An I2I Inpainting Approach for Efficient Channel Knowledge Map Construction 14 Jun 2024 · 0 repositories · arXiv:2406.09822
-
EulerMormer: Robust Eulerian Motion Magnification via Dynamic Filtering within Transformer 7 Dec 2023 · 2 repositories · arXiv:2312.04152
-
CAP: Correlation-Aware Pruning for Highly-Accurate Sparse Vision Models 14 Oct 2022 · 0 repositories · arXiv:2210.09223
-
SimA: Simple Softmax-free Attention for Vision Transformers 17 Jun 2022 · 1 repository · arXiv:2206.08898Syntology ran 0 of 3 samples · 3 unverified
-
PE-former: Pose Estimation Transformer 9 Dec 2021 · 1 repository · arXiv:2112.04981
-
XCiT: Cross-Covariance Image Transformers 17 Jun 2021 · 12 repositories · arXiv:2106.09681Syntology ran 3 of 14 samples · 11 unverified · 3 pointer-only (licence)
Tasks archive 2025-07-28
16 tasks the archive attaches to papers tagged with this method, by distinct papers. A task without a page in the catalog is plain text.
Usage over time archive 2025-07-28
Components: the archive holds no method-to-method composition, so PwC's Components table cannot be rebuilt; the Papers list carries no Results column for the same reason (the archive does not join its leaderboard rows to method tags).
Categories archive 2025-07-28
Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections