{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/ernie-enhanced-language-representation-with","title":"ERNIE: Enhanced Language Representation with Informative Entities","arxiv_id":"1905.07129","date":"2019-05-17","proceeding":"ACL 2019 7","authors":["Zhengyan Zhang","Xu Han","Zhiyuan Liu","Xin Jiang","Maosong Sun","Qun Liu"],"abstract":"Neural language representation models such as BERT pre-trained on large-scale corpora can well capture rich semantic patterns from plain text, and be fine-tuned to consistently improve the performance of various NLP tasks. However, the existing pre-trained language models rarely consider incorporating knowledge graphs (KGs), which can provide rich structured knowledge facts for better language understanding. We argue that informative entities in KGs can enhance language representation with external knowledge. In this paper, we utilize both large-scale textual corpora and KGs to train an enhanced language representation model (ERNIE), which can take full advantage of lexical, syntactic, and knowledge information simultaneously. The experimental results have demonstrated that ERNIE achieves significant improvements on various knowledge-driven tasks, and meanwhile is comparable with the state-of-the-art model BERT on other common NLP tasks. The source code of this paper can be obtained from https://github.com/thunlp/ERNIE.","url_abs":"https://arxiv.org/abs/1905.07129v3","url_pdf":"https://arxiv.org/pdf/1905.07129v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"ernie-enhanced-language-representation-with","repo_url":"https://github.com/thunlp/ERNIE","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"ernie-enhanced-language-representation-with","repo_url":"https://github.com/Mind23-2/MindCode-136","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"mindspore","reach":{"status":"ok"}}],"tasks":[{"task_slug":"entity-linking","task_name":"Entity Linking"},{"task_slug":"entity-typing","task_name":"Entity Typing"},{"task_slug":"knowledge-graphs","task_name":"Knowledge Graphs"},{"task_slug":"linguistic-acceptability","task_name":"Linguistic Acceptability"},{"task_slug":"natural-language-inference","task_name":"Natural Language Inference"},{"task_slug":"paraphrase-identification","task_name":"Paraphrase Identification"},{"task_slug":"relation-classification","task_name":"Relation Classification"},{"task_slug":"relation-extraction","task_name":"Relation Extraction"},{"task_slug":"semantic-textual-similarity","task_name":"Semantic Textual Similarity"},{"task_slug":"sentiment-analysis","task_name":"Sentiment Analysis"}],"methods":[{"method_slug":"adam","method_name":"Adam"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"attention-dropout","method_name":"Attention Dropout"},{"method_slug":"bert","method_name":"BERT"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"ernie","method_name":"ERNIE"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"linear-warmup-with-linear-decay","method_name":"Linear Warmup With Linear Decay"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"weight-decay","method_name":"Weight Decay"},{"method_slug":"wordpiece","method_name":"WordPiece"}],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/entity-linking-on-figer","task":"Entity Linking","dataset":"FIGER","model":"ERNIE","rank_in_archive_order":1,"of":1,"metrics":{"Accuracy":"57.19","Macro F1":"76.51","Micro F1":"73.39"},"uses_additional_data":false},{"leaderboard":"/sota/entity-typing-on-open-entity","task":"Entity Typing","dataset":"Open Entity","model":"ERNIE","rank_in_archive_order":3,"of":3,"metrics":{"F1":"75.56","Precision":"78.42","Recall":"72.9"},"uses_additional_data":false},{"leaderboard":"/sota/linguistic-acceptability-on-cola","task":"Linguistic Acceptability","dataset":"CoLA","model":"ERNIE","rank_in_archive_order":35,"of":43,"metrics":{"Accuracy":"52.3%"},"uses_additional_data":false},{"leaderboard":"/sota/natural-language-inference-on-multinli","task":"Natural Language Inference","dataset":"MultiNLI","model":"ERNIE","rank_in_archive_order":34,"of":67,"metrics":{"Matched":"84.0","Mismatched":"83.2"},"uses_additional_data":false},{"leaderboard":"/sota/natural-language-inference-on-qnli","task":"Natural Language Inference","dataset":"QNLI","model":"ERNIE","rank_in_archive_order":29,"of":43,"metrics":{"Accuracy":"91.3%"},"uses_additional_data":false},{"leaderboard":"/sota/natural-language-inference-on-rte","task":"Natural Language Inference","dataset":"RTE","model":"ERNIE","rank_in_archive_order":59,"of":90,"metrics":{"Accuracy":"68.8%"},"uses_additional_data":false},{"leaderboard":"/sota/paraphrase-identification-on-quora-question","task":"Paraphrase Identification","dataset":"Quora Question Pairs","model":"ERNIE","rank_in_archive_order":15,"of":31,"metrics":{"F1":"71.2"},"uses_additional_data":false},{"leaderboard":"/sota/relation-classification-on-tacred-1","task":"Relation Classification","dataset":"TACRED","model":"BERT","rank_in_archive_order":4,"of":17,"metrics":{"F1":"66.0"},"uses_additional_data":false},{"leaderboard":"/sota/relation-classification-on-tacred-1","task":"Relation Classification","dataset":"TACRED","model":"ERNIE","rank_in_archive_order":6,"of":17,"metrics":{"F1":"68.0"},"uses_additional_data":false},{"leaderboard":"/sota/relation-extraction-on-fewrel","task":"Relation Extraction","dataset":"FewRel","model":"ERNIE","rank_in_archive_order":2,"of":2,"metrics":{"F1":"88.32","Precision":"88.49","Recall":"88.44"},"uses_additional_data":false},{"leaderboard":"/sota/relation-extraction-on-tacred","task":"Relation Extraction","dataset":"TACRED","model":"ERNIE","rank_in_archive_order":28,"of":40,"metrics":{"F1":"67.97"},"uses_additional_data":false},{"leaderboard":"/sota/semantic-textual-similarity-on-mrpc","task":"Semantic Textual Similarity","dataset":"MRPC","model":"ERNIE","rank_in_archive_order":21,"of":45,"metrics":{"Accuracy":"88.2%"},"uses_additional_data":false},{"leaderboard":"/sota/semantic-textual-similarity-on-sts-benchmark","task":"Semantic Textual Similarity","dataset":"STS Benchmark","model":"ERNIE","rank_in_archive_order":26,"of":66,"metrics":{"Pearson Correlation":"0.832"},"uses_additional_data":false},{"leaderboard":"/sota/sentiment-analysis-on-sst-2-binary","task":"Sentiment Analysis","dataset":"SST-2 Binary classification","model":"ERNIE","rank_in_archive_order":40,"of":87,"metrics":{"Accuracy":"93.5"},"uses_additional_data":false}],"syntology":{"syntology_url":null,"atlas_url":"https://app.syntology.ai/?focus=1905.07129","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"1905.07129"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/Mind23-2/MindCode-136","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/thunlp/ERNIE","reach":null}],"summary":{"ran_honours":2,"ran_draft_wrong":1},"by_repo_kind":{"official":{"samples":2,"ran":2,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":1,"samples":[{"code_sha256_prefix":"eb725d5794b15f6b","entry":"accuracy","repo":null,"repo_kind":null,"path":null,"file_url":null,"link_basis":"identical_code_first_harvested_elsewhere","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":null,"inline_ok":false,"mcp_get_code":{"code_sha256":"eb725d5794b15f6b"}},{"code_sha256_prefix":"c248635dec6d8a67","entry":"convert_examples_to_features","repo":"thunlp/ERNIE","repo_kind":"official","path":"code/run_fewrel.py","file_url":"https://github.com/thunlp/ERNIE/blob/HEAD/code/run_fewrel.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c248635dec6d8a67"}},{"code_sha256_prefix":"43676c538c76c491","entry":"warmup_linear","repo":"thunlp/ERNIE","repo_kind":"official","path":"code/run_pretrain.py","file_url":"https://github.com/thunlp/ERNIE/blob/HEAD/code/run_pretrain.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"43676c538c76c491"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}