{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/alternative-weighting-schemes-for-elmo","title":"Alternative Weighting Schemes for ELMo Embeddings","arxiv_id":"1904.02954","date":"2019-04-05","proceeding":null,"authors":["Nils Reimers","Iryna Gurevych"],"abstract":"ELMo embeddings (Peters et. al, 2018) had a huge impact on the NLP community\nand may recent publications use these embeddings to boost the performance for\ndownstream NLP tasks. However, integration of ELMo embeddings in existent NLP\narchitectures is not straightforward. In contrast to traditional word\nembeddings, like GloVe or word2vec embeddings, the bi-directional language\nmodel of ELMo produces three 1024 dimensional vectors per token in a sentence.\nPeters et al. proposed to learn a task-specific weighting of these three\nvectors for downstream tasks. However, this proposed weighting scheme is not\nfeasible for certain tasks, and, as we will show, it does not necessarily yield\noptimal performance. We evaluate different methods that combine the three\nvectors from the language model in order to achieve the best possible\nperformance in downstream NLP tasks. We notice that the third layer of the\npublished language model often decreases the performance. By learning a\nweighted average of only the first two layers, we are able to improve the\nperformance for many datasets. Due to the reduced complexity of the language\nmodel, we have a training speed-up of 19-44% for the downstream task.","url_abs":"http://arxiv.org/abs/1904.02954v1","url_pdf":"http://arxiv.org/pdf/1904.02954v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"alternative-weighting-schemes-for-elmo","repo_url":"https://github.com/UKPLab/elmo-bilstm-cnn-crf","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"tf","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"language-modeling","task_name":"Language Modeling"},{"task_slug":"language-modelling","task_name":"Language Modelling"},{"task_slug":"sentence","task_name":"Sentence"},{"task_slug":"word-embeddings","task_name":"Word Embeddings"}],"methods":[{"method_slug":"bilstm","method_name":"BiLSTM"},{"method_slug":"elmo","method_name":"ELMo"},{"method_slug":"glove","method_name":"GloVe"},{"method_slug":"lstm","method_name":"LSTM"},{"method_slug":"sigmoid-activation","method_name":"Sigmoid Activation"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"tanh-activation","method_name":"Tanh Activation"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/1904.02954","atlas_url":"https://app.syntology.ai/?focus=1904.02954","mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}