{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/adaptation-of-deep-bidirectional-multilingual","title":"Adaptation of Deep Bidirectional Multilingual Transformers for Russian Language","arxiv_id":"1905.07213","date":"2019-05-17","proceeding":null,"authors":["Yuri Kuratov","Mikhail Arkhipov"],"abstract":"The paper introduces methods of adaptation of multilingual masked language models for a specific language. Pre-trained bidirectional language models show state-of-the-art performance on a wide range of tasks including reading comprehension, natural language inference, and sentiment analysis. At the moment there are two alternative approaches to train such models: monolingual and multilingual. While language specific models show superior performance, multilingual models allow to perform a transfer from one language to another and solve tasks for different languages simultaneously. This work shows that transfer learning from a multilingual model to monolingual model results in significant growth of performance on such tasks as reading comprehension, paraphrase detection, and sentiment analysis. Furthermore, multilingual initialization of monolingual model substantially reduces training time. Pre-trained models for the Russian language are open sourced.","url_abs":"https://arxiv.org/abs/1905.07213v1","url_pdf":"https://arxiv.org/pdf/1905.07213v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"adaptation-of-deep-bidirectional-multilingual","repo_url":"https://github.com/fulstock/mrc_nested_ner_ru","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}},{"paper_slug":"adaptation-of-deep-bidirectional-multilingual","repo_url":"https://github.com/deepmipt/bert/tree/feat/multi_gpu","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"tf","reach":null}],"tasks":[{"task_slug":"natural-language-inference","task_name":"Natural Language Inference"},{"task_slug":"paraphrase-identification","task_name":"Paraphrase Identification"},{"task_slug":"question-answering","task_name":"Question Answering"},{"task_slug":"reading-comprehension","task_name":"Reading Comprehension"},{"task_slug":"sentiment-analysis","task_name":"Sentiment Analysis"},{"task_slug":"transfer-learning","task_name":"Transfer Learning"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/question-answering-on-squad11","task":"Question Answering","dataset":"SQuAD1.1","model":"RuBERT","rank_in_archive_order":213,"of":213,"metrics":{"F1":"84.6"},"uses_additional_data":false},{"leaderboard":"/sota/sentiment-analysis-on-rusentiment","task":"Sentiment Analysis","dataset":"RuSentiment","model":"RuBERT","rank_in_archive_order":3,"of":3,"metrics":{"Weighted F1":"72.63"},"uses_additional_data":false}],"syntology":{"syntology_url":"https://syntology.ai/paper/1905.07213","atlas_url":"https://app.syntology.ai/?focus=1905.07213","mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}