Methods › Natural Language Processing › Sentence Embeddings › TSDAE
TSDAE
Introduced by Kexin Wang et al. in TSDAE: Using Transformer-based Sequential Denoising Auto-Encoder for Unsupervised Sentence Embedding Learning
archive 2025-07-28 Description, source and code snippet are the archive's method entry.
TSDAE is an unsupervised sentence embedding method. During training, TSDAE encodes corrupted sentences into fixed-sized vectors and requires the decoder to reconstruct the original sentences from this sentence embedding. For good reconstruction quality, the semantics must be captured well in the sentence embedding from the encoder. Later, at inference, we only use the encoder for creating sentence embeddings.
The model architecture of TSDAE is a modified encoder-decoder Transformer where the key and value of the cross-attention are both confined to the sentence embedding only. Formally, the formulation of the modified cross-attention is:
H⁽ᵏ⁾= Attention (H⁽ᵏ⁻¹⁾,[sᵀ],[sᵀ])
Attention(Q, K, V)=softmax((Q Kᵀ)/(√(d))) V
where H⁽ᵏ⁾ ∈ℝ^(t ×d) is the decoder hidden states within t decoding steps at the k-th layer, d is the size of the sentence embedding, [sᵀ] ∈ℝ^(1 ×d) is a one-row matrix including the sentence embedding vector and Q, K and V are the query, key and value, respectively. By exploring different configurations on the STS benchmark dataset, the authors discover that the best combination is: (1) adopting deletion as the input noise and setting the deletion ratio to 0.6,(2) using the output of the [CLS] token as fixed-sized sentence representation (3) tying the encoder and decoder parameters during training.
Papers archive 2025-07-28
3 shown of 3, newest first. Repository counts are the archive's code-links table. A Syntology line states what Syntology ran from that paper's harvested code; it is per sample and not a correctness claim.
-
GPL: Generative Pseudo Labeling for Unsupervised Domain Adaptation of Dense Retrieval 16 Jan 2022 · 0 repositories
-
GPL: Generative Pseudo Labeling for Unsupervised Domain Adaptation of Dense Retrieval 14 Dec 2021 · 5 repositories · arXiv:2112.07577Syntology ran 0 of 5 samples · 5 unverified
-
TSDAE: Using Transformer-based Sequential Denoising Auto-Encoder for Unsupervised Sentence Embedding Learning 14 Apr 2021 · 6 repositories · arXiv:2104.06979Syntology ran 0 of 4 samples · 4 unverified
Tasks archive 2025-07-28
16 tasks the archive attaches to papers tagged with this method, by distinct papers. A task without a page in the catalog is plain text.
Usage over time archive 2025-07-28
Components: the archive holds no method-to-method composition, so PwC's Components table cannot be rebuilt; the Papers list carries no Results column for the same reason (the archive does not join its leaderboard rows to method tags).
Categories archive 2025-07-28
Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections