{"url":"/method/tsdae","slug":"tsdae","name":"TSDAE","full_name":"TSDAE","full_name_withheld":false,"description_markdown":"**TSDAE** is an unsupervised sentence embedding method. During training, TSDAE encodes corrupted sentences into fixed-sized vectors and requires the decoder to reconstruct the original sentences from this sentence embedding. For good reconstruction quality, the semantics must be captured well in the sentence embedding from the encoder. Later, at inference, we only use the encoder for creating sentence embeddings.\r\n\r\nThe model architecture of TSDAE is a modified [encoder-decoder Transformer](https://paperswithcode.com/methods/category/autoencoding-transformers) where the key and value of the cross-attention are both confined to the sentence embedding only. Formally, the formulation of the modified cross-attention is:\r\n\r\n$$\r\nH^{(k)}=\\text { Attention }\\left(H^{(k-1)},\\left[s^{T}\\right],\\left[s^{T}\\right]\\right)\r\n$$\r\n\r\n$$\r\n\\operatorname{Attention}(Q, K, V)=\\operatorname{softmax}\\left(\\frac{Q K^{T}}{\\sqrt{d}}\\right) V\r\n$$\r\n\r\nwhere $H^{(k)} \\in \\mathbb{R}^{t \\times d}$ is the decoder hidden states within $t$ decoding steps at the $k$-th layer, $d$ is the size of the sentence embedding, $\\left[s^{T}\\right] \\in \\mathbb{R}^{1 \\times d}$ is a one-row matrix including the sentence embedding vector and $Q, K$ and $V$ are the query, key and value, respectively. By exploring different configurations on the STS benchmark dataset, the authors discover that the best combination is: (1) adopting deletion as the input noise and setting the deletion ratio to $0.6,(2)$ using the output of the [CLS] token as fixed-sized sentence representation (3) tying the encoder and decoder parameters during training.","description_state":"present","introduced_year":null,"introduced_by":{"title":"TSDAE: Using Transformer-based Sequential Denoising Auto-Encoder for Unsupervised Sentence Embedding Learning","paper":"/paper/tsdae-using-transformer-based-sequential","first_author":"Kexin Wang","n_authors":3,"url_abs":null,"archive_paper_url":"https://paperswithcode.com/paper/tsdae-using-transformer-based-sequential"},"source":{"url":"https://arxiv.org/abs/2104.06979v3","title":"TSDAE: Using Transformer-based Sequential Denoising Auto-Encoder for Unsupervised Sentence Embedding Learning","url_on_a_paper_host":true},"code_snippet_url":null,"code_snippet_url_on_a_code_host":false,"categories":[{"area":"Natural Language Processing","area_id":"natural-language-processing","collection":"Sentence Embeddings","url":"/methods/category/sentence-embeddings","pwc_aliases":[]}],"n_papers_tagged":3,"archive_num_papers":3,"papers_newest_first":[{"paper":null,"title":"GPL: Generative Pseudo Labeling for Unsupervised Domain Adaptation of Dense Retrieval","date":"2022-01-16","arxiv_id":null,"n_code_links":0,"syntology":null},{"paper":"/paper/gpl-generative-pseudo-labeling-for","title":"GPL: Generative Pseudo Labeling for Unsupervised Domain Adaptation of Dense Retrieval","date":"2021-12-14","arxiv_id":"2112.07577","n_code_links":5,"syntology":{"ran":0,"of":5,"unverified":5,"pointer_only":0}},{"paper":"/paper/tsdae-using-transformer-based-sequential","title":"TSDAE: Using Transformer-based Sequential Denoising Auto-Encoder for Unsupervised Sentence Embedding Learning","date":"2021-04-14","arxiv_id":"2104.06979","n_code_links":6,"syntology":{"ran":0,"of":4,"unverified":4,"pointer_only":0}}],"papers_shown":3,"tasks":[{"task":"/task/domain-adaptation","name":"Domain Adaptation","papers":3},{"task":"/task/retrieval","name":"Retrieval","papers":2},{"task":"/task/unsupervised-domain-adaptation","name":"Unsupervised Domain Adaptation","papers":2},{"task":"/task/denoising","name":"Denoising","papers":1},{"task":"/task/information-retrieval","name":"Information Retrieval","papers":1},{"task":"/task/language-modeling","name":"Language Modeling","papers":1},{"task":"/task/language-modelling","name":"Language Modelling","papers":1},{"task":"/task/paraphrase-identification","name":"Paraphrase Identification","papers":1},{"task":"/task/re-ranking","name":"Re-Ranking","papers":1},{"task":"/task/sts","name":"STS","papers":1},{"task":"/task/semantic-textual-similarity","name":"Semantic Textual Similarity","papers":1},{"task":"/task/sentence","name":"Sentence","papers":1},{"task":"/task/sentence-embedding","name":"Sentence Embedding","papers":1},{"task":"/task/sentence-embeddings","name":"Sentence Embeddings","papers":1},{"task":"/task/sentence-embedding-1","name":"Sentence-Embedding","papers":1},{"task":"/task/zero-shot-text-search","name":"Zero-shot Text Search","papers":1}],"tasks_shown":16,"n_tasks":16,"usage_by_year":[{"year":"2021","papers":2},{"year":"2022","papers":1}],"row_source":"methods_table","archive":{"source":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","archive_url":"https://paperswithcode.com/method/tsdae"},"syntology_read_at":"2026-09-24T18:15:14+00:00"}