{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/evaluation-of-sentence-embeddings-in","title":"Evaluation of sentence embeddings in downstream and linguistic probing tasks","arxiv_id":"1806.06259","date":"2018-06-16","proceeding":null,"authors":["Christian S. Perone","Roberto Silveira","Thomas S. Paula"],"abstract":"Despite the fast developmental pace of new sentence embedding methods, it is\nstill challenging to find comprehensive evaluations of these different\ntechniques. In the past years, we saw significant improvements in the field of\nsentence embeddings and especially towards the development of universal\nsentence encoders that could provide inductive transfer to a wide variety of\ndownstream tasks. In this work, we perform a comprehensive evaluation of recent\nmethods using a wide variety of downstream and linguistic feature probing\ntasks. We show that a simple approach using bag-of-words with a recently\nintroduced language model for deep context-dependent word embeddings proved to\nyield better results in many tasks when compared to sentence encoders trained\non entailment datasets. We also show, however, that we are still far away from\na universal encoder that can perform consistently across several downstream\ntasks.","url_abs":"http://arxiv.org/abs/1806.06259v1","url_pdf":"http://arxiv.org/pdf/1806.06259v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"evaluation-of-sentence-embeddings-in","repo_url":"https://github.com/allenai/bilm-tf","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"tf","reach":{"status":"ok","spdx":"Apache-2.0"}},{"paper_slug":"evaluation-of-sentence-embeddings-in","repo_url":"https://github.com/cheng18/bilm-tf","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"tf","reach":{"status":"ok","spdx":"Apache-2.0"}},{"paper_slug":"evaluation-of-sentence-embeddings-in","repo_url":"https://github.com/horizonheart/ELMO","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"tf","reach":{"status":"ok","spdx":"Apache-2.0"}},{"paper_slug":"evaluation-of-sentence-embeddings-in","repo_url":"https://github.com/jvdbogae/artverc","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok","spdx":"Apache-2.0"}},{"paper_slug":"evaluation-of-sentence-embeddings-in","repo_url":"https://github.com/kunde122/bilm-tf","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"tf","reach":{"status":"ok","spdx":"Apache-2.0"}},{"paper_slug":"evaluation-of-sentence-embeddings-in","repo_url":"https://github.com/mingdachen/bilm-tf","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"tf","reach":null},{"paper_slug":"evaluation-of-sentence-embeddings-in","repo_url":"https://github.com/nlp-research/bilm-tf","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"tf","reach":{"status":"ok"}},{"paper_slug":"evaluation-of-sentence-embeddings-in","repo_url":"https://github.com/seunghwan1228/ELMO","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"tf","reach":{"status":"ok","spdx":"Apache-2.0"}},{"paper_slug":"evaluation-of-sentence-embeddings-in","repo_url":"https://github.com/shaneding/bilm-tf-experimentation","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"tf","reach":null},{"paper_slug":"evaluation-of-sentence-embeddings-in","repo_url":"https://github.com/shelleyHLX/bilm_EMLo","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"tf","reach":null},{"paper_slug":"evaluation-of-sentence-embeddings-in","repo_url":"https://github.com/sidak/SentEval","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"evaluation-of-sentence-embeddings-in","repo_url":"https://github.com/weixsong/bilm-tf","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"tf","reach":null},{"paper_slug":"evaluation-of-sentence-embeddings-in","repo_url":"https://github.com/yangzonglin1994/bilm-tf-extended","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"tf","reach":{"status":"ok","spdx":"Apache-2.0"}},{"paper_slug":"evaluation-of-sentence-embeddings-in","repo_url":"https://github.com/young-zonglin/bilm-tf-extended","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"tf","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"language-modeling","task_name":"Language Modeling"},{"task_slug":"language-modelling","task_name":"Language Modelling"},{"task_slug":"sentence","task_name":"Sentence"},{"task_slug":"sentence-embedding","task_name":"Sentence Embedding"},{"task_slug":"sentence-embeddings","task_name":"Sentence Embeddings"},{"task_slug":"sentence-embedding-1","task_name":"Sentence-Embedding"},{"task_slug":"word-embeddings","task_name":"Word Embeddings"}],"methods":[{"method_slug":"1x1-convolution","method_name":"1x1 Convolution"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/1806.06259","atlas_url":"https://app.syntology.ai/?focus=1806.06259","mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}