Methods › Natural Language Processing › Sentence Embeddings › TSDAE

TSDAE

3 papers tagged archive 2025-07-28

Introduced by Kexin Wang et al. in TSDAE: Using Transformer-based Sequential Denoising Auto-Encoder for Unsupervised Sentence Embedding Learning

archive 2025-07-28 Description, source and code snippet are the archive's method entry.

TSDAE is an unsupervised sentence embedding method. During training, TSDAE encodes corrupted sentences into fixed-sized vectors and requires the decoder to reconstruct the original sentences from this sentence embedding. For good reconstruction quality, the semantics must be captured well in the sentence embedding from the encoder. Later, at inference, we only use the encoder for creating sentence embeddings.

The model architecture of TSDAE is a modified encoder-decoder Transformer where the key and value of the cross-attention are both confined to the sentence embedding only. Formally, the formulation of the modified cross-attention is:

H⁽ᵏ⁾= Attention (H⁽ᵏ⁻¹⁾,[sᵀ],[sᵀ])

Attention(Q, K, V)=softmax((Q Kᵀ)/(√(d))) V

where H⁽ᵏ⁾ ∈ℝ^(t ×d) is the decoder hidden states within t decoding steps at the k-th layer, d is the size of the sentence embedding, [sᵀ] ∈ℝ^(1 ×d) is a one-row matrix including the sentence embedding vector and Q, K and V are the query, key and value, respectively. By exploring different configurations on the STS benchmark dataset, the authors discover that the best combination is: (1) adopting deletion as the input noise and setting the deletion ratio to 0.6,(2) using the output of the [CLS] token as fixed-sized sentence representation (3) tying the encoder and decoder parameters during training.

PaperSource

In Syntology about another paper, arXiv:2607.24542: repository known, no verified implementation (declared_code; repositories: 1 listed of 1 in the graph, strongest link: listed only). About another paper: this answer is about arXiv:2607.24542, while the archive credits arXiv:2104.06979.

From the lookup's note on this name: The same name also belongs to the paper “TSDAE: Using Transformer-based Sequential Denoising Auto-Encoder for Unsupervised Sentence Embedding Learning” (arXiv:2104.06979, 243 citations), which the lookup finds cited far more than the paper this answer is about (0 citations), so this answer may be about a different sense of the name. For arXiv:2104.06979: its page on this site · ask the lookup about it.

Origin paper of this answer, arXiv:2607.24542 (its page on this site · its page on Syntology): in Syntology's graph layer with no harvested code sample.

Syntology's have check for this name (free, it answers coverage only), read 2026-10-05; ask it again live. A level is the served verification level of that one answer; nothing here is summed across methods.

Papers archive 2025-07-28

archive papers tagged: 3 · with a code link: 2 · where Syntology ran a sample: 2 (2 with a run with no instrument failure, 0 where every run was a failure of Syntology's instrument) Syntology

Show: all tagged papersonly where code ran (2 of 3 tagged: 2 with a run with no instrument failure, 0 where every run was a failure of Syntology's instrument)

Page 1 of 1: papers 1 to 3 of 3, newest first by the archive's date (ties by slug), in archive order.

Repository counts are the archive's code-links table. A Syntology line states what Syntology ran from that paper's harvested code, as “N ran (of which C constructed an object rather than computing a result; K with no instrument failure: H honoured, V violated, P with no contract checked; I where Syntology's instrument failed) · U unverified”; the instrument figure counts failures of Syntology's instrument, not of the code. It is per sample and not a correctness claim. When the archive marks a repository official for the paper, the line starts with that repository's state (the archive's flag, not a verdict on who wrote the code; “community repositories only” when every sample that ran came from a community repository, “official: no sample here; runs from other or unrecorded repositories” when some came from a repository the paper names or has in its text, or from none recorded); hover it for the repositories the samples that ran came from.

Tasks archive 2025-07-28

16 tasks the archive attaches to papers tagged with this method, by distinct papers. A task without a page in the catalog is plain text.

TaskPapers
Domain Adaptation3
Retrieval2
Unsupervised Domain Adaptation2
Denoising1
Information Retrieval1
Language Modeling1
Language Modelling1
Paraphrase Identification1
Re-Ranking1
STS1
Semantic Textual Similarity1
Sentence1
Sentence Embedding1
Sentence Embeddings1
Sentence-Embedding1
Zero-shot Text Search1

Usage over time archive 2025-07-28

Papers per year tagged with TSDAE: 2021 to 2022, peak 2 2 0 2021: 2 papers 2021 2022: 1 paper 2022
Papers per year the archive tags with this method, by the paper's archive date (3 dated). Bars are counts, not a trend claim.

Components: the archive holds no method-to-method composition, so PwC's Components table cannot be rebuilt; the Papers list carries no Results column for the same reason (the archive does not join its leaderboard rows to method tags).

Categories archive 2025-07-28

Sentence Embeddings

Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections