Papers › CLEAR: Contrastive Learning for Sentence Representation

CLEAR: Contrastive Learning for Sentence Representation

31 Dec 2020arXiv:2012.15466archive 2025-07-28

Zhuofeng Wu, Sinong Wang, Jiatao Gu, Madian Khabsa, Fei Sun, Hao Ma

Pre-trained language models have proven their unique powers in capturing implicit language features. However, most pre-training approaches focus on the word-level training objective, while sentence-level objectives are rarely studied. In this paper, we propose Contrastive LEArning for sentence Representation (CLEAR), which employs multiple sentence-level augmentation strategies in order to learn a noise-invariant sentence representation. These augmentations include word and span deletion, reordering, and substitution. Furthermore, we investigate the key reasons that make contrastive learning effective through numerous experiments. We observe that different sentence augmentations during pre-training lead to different performance improvements on various downstream tasks. Our approach is shown to outperform multiple existing methods on both SentEval and GLUE benchmarks.

Paper PDF

In Syntology Open this paper in Syntology's Atlas, the map of the papers in Syntology's graph and their citations.

Code

No code repository is listed for this paper in the archive or in Syntology's graph.

Code Syntology ran Syntology

Not run by Syntology. Nothing on this page verifies that the listed code works.

Tasks

Contrastive Learning Linguistic Acceptability Natural Language Inference Question Answering Semantic Textual Similarity Sentence Sentiment Analysis

Results from the paper archive 2025-07-28

Task	Dataset	Model	Metric	Value	Rank at snapshot	Leaderboard	Report
Linguistic Acceptability	CoLA	MLM+ del-span+ reorder	Accuracy	64.3%	#26 of 43	Archive leaderboard	report
Natural Language Inference	QNLI	MLM+ subs+ del-span	Accuracy	93.4%	#21 of 43	Archive leaderboard	report
Natural Language Inference	RTE	MLM+ del-span	Accuracy	79.8%	#37 of 90	Archive leaderboard	report
Question Answering	Quora Question Pairs	MLM+ subs+ del-span	Accuracy	90.3%	#5 of 19	Archive leaderboard	report
Semantic Textual Similarity	MRPC	MLM+ del-word+ reorder	Accuracy	90.6%	#11 of 45	Archive leaderboard	report
Semantic Textual Similarity	STS Benchmark	MLM+ del-word	Pearson Correlation	0.905	#18 of 66	Archive leaderboard	report
Sentiment Analysis	SST-2 Binary classification	MLM+ del-word+ reorder	Accuracy	94.5	#35 of 87	Archive leaderboard	report

Ranks are positions in the archive's leaderboards as they stood at the 2025-07-28 snapshot. Results published since then are not among these rows, so a rank here is not a current standing.

Methods

Contrastive Learning

Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections