Papers › Evaluating Contextualized Embeddings on 54 Languages in POS Tagging, Lemmatization and...

Evaluating Contextualized Embeddings on 54 Languages in POS Tagging, Lemmatization and Dependency Parsing

20 Aug 2019arXiv:1908.07448archive 2025-07-28

Milan Straka, Jana Straková, Jan Hajič

We present an extensive evaluation of three recently proposed methods for contextualized embeddings on 89 corpora in 54 languages of the Universal Dependencies 2.3 in three tasks: POS tagging, lemmatization, and dependency parsing. Employing the BERT, Flair and ELMo as pretrained embedding inputs in a strong baseline of UDPipe 2.0, one of the best-performing systems of the CoNLL 2018 Shared Task and an overall winner of the EPE 2018, we present a one-to-one comparison of the three contextualized word embedding methods, as well as a comparison with word2vec-like pretrained embeddings and with end-to-end character-level word embeddings. We report state-of-the-art results in all three tasks as compared to results on UD 2.2 in the CoNLL 2018 Shared Task.

Paper PDF

In Syntology Open this paper in Syntology's Atlas, the map of the papers in Syntology's graph and their citations.

Code

No code repository is listed for this paper in the archive or in Syntology's graph.

Code Syntology ran Syntology

Not run by Syntology. Nothing on this page verifies that the listed code works.

Tasks

Dependency Parsing Lemmatization POS POS Tagging Word Embeddings

Results from the paper archive 2025-07-28

Task	Dataset	Model	Metric	Value	Rank at snapshot	Leaderboard	Report
Dependency Parsing	Universal Dependencies	UDPipe 2.0 + mBERT + FLAIR	LAS	84.60	#1 of 6	Archive leaderboard	report
Dependency Parsing	Universal Dependencies	UDPipe 2.0 + mBERT + FLAIR	UAS	87.64	#1 of 6	Archive leaderboard	report

Ranks are positions in the archive's leaderboards as they stood at the 2025-07-28 snapshot. Results published since then are not among these rows, so a rank here is not a current standing.

Methods

Adam Attention Attention Dropout BERT BiLSTM Dense Connections Dropout ELMo LSTM Layer Normalization Linear Layer Linear Warmup With Linear Decay Multi-Head Attention Residual Connection Sigmoid Activation Softmax Tanh Activation Weight Decay WordPiece

Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections