{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/entities-as-experts-sparse-memory-access-with","title":"Entities as Experts: Sparse Memory Access with Entity Supervision","arxiv_id":"2004.07202","date":"2020-04-15","proceeding":"EMNLP 2020 11","authors":["Thibault Févry","Livio Baldini Soares","Nicholas FitzGerald","Eunsol Choi","Tom Kwiatkowski"],"abstract":"We focus on the problem of capturing declarative knowledge about entities in the learned parameters of a language model. We introduce a new model - Entities as Experts (EAE) - that can access distinct memories of the entities mentioned in a piece of text. Unlike previous efforts to integrate entity knowledge into sequence models, EAE's entity representations are learned directly from text. We show that EAE's learned representations capture sufficient knowledge to answer TriviaQA questions such as \"Which Dr. Who villain has been played by Roger Delgado, Anthony Ainley, Eric Roberts?\", outperforming an encoder-generator Transformer model with 10x the parameters. According to the LAMA knowledge probes, EAE contains more factual knowledge than a similarly sized BERT, as well as previous approaches that integrate external sources of entity knowledge. Because EAE associates parameters with specific entities, it only needs to access a fraction of its parameters at inference time, and we show that the correct identification and representation of entities is essential to EAE's performance.","url_abs":"https://arxiv.org/abs/2004.07202v2","url_pdf":"https://arxiv.org/pdf/2004.07202v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"entities-as-experts-sparse-memory-access-with","repo_url":"https://github.com/erolm-a/colla-framework","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"language-modeling","task_name":"Language Modeling"},{"task_slug":"language-modelling","task_name":"Language Modelling"},{"task_slug":"triviaqa","task_name":"TriviaQA"}],"methods":[{"method_slug":"absolute-position-encodings","method_name":"Absolute Position Encodings"},{"method_slug":"adam","method_name":"Adam"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"attention-dropout","method_name":"Attention Dropout"},{"method_slug":"bert","method_name":"BERT"},{"method_slug":"bpe","method_name":"BPE"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"label-smoothing","method_name":"Label Smoothing"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"linear-warmup-with-linear-decay","method_name":"Linear Warmup With Linear Decay"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"position-wise-feed-forward-layer","method_name":"Position-Wise Feed-Forward Layer"},{"method_slug":"relu","method_name":"ReLU"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"transformer","method_name":"Transformer"},{"method_slug":"weight-decay","method_name":"Weight Decay"},{"method_slug":"wordpiece","method_name":"WordPiece"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2004.07202","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2004.07202"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/erolm-a/colla-framework","reach":null}],"summary":{"ran":11,"unverified":3},"by_repo_kind":{"listed":{"samples":14,"ran":11,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":14,"samples":[{"code_sha256_prefix":"9aac70e1d1b8c20d","entry":"BioClassifier","repo":"erolm-a/colla-framework","repo_kind":"listed","path":"src/models/entities_as_experts.py","file_url":"https://github.com/erolm-a/colla-framework/blob/HEAD/src/models/entities_as_experts.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"9aac70e1d1b8c20d"}},{"code_sha256_prefix":"cf73b2167981e1af","entry":"EaELinearWithLayerNorm","repo":"erolm-a/colla-framework","repo_kind":"listed","path":"src/models/entities_as_experts.py","file_url":"https://github.com/erolm-a/colla-framework/blob/HEAD/src/models/entities_as_experts.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"cf73b2167981e1af"}},{"code_sha256_prefix":"99e7bea3063859c4","entry":"EaEPredictionHead","repo":"erolm-a/colla-framework","repo_kind":"listed","path":"src/models/entities_as_experts.py","file_url":"https://github.com/erolm-a/colla-framework/blob/HEAD/src/models/entities_as_experts.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"99e7bea3063859c4"}},{"code_sha256_prefix":"55b3f7e6ffe06fbf","entry":"EntityMemory","repo":"erolm-a/colla-framework","repo_kind":"listed","path":"src/models/entities_as_experts.py","file_url":"https://github.com/erolm-a/colla-framework/blob/HEAD/src/models/entities_as_experts.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"55b3f7e6ffe06fbf"}},{"code_sha256_prefix":"8dd59405e934df62","entry":"EntityPred","repo":"erolm-a/colla-framework","repo_kind":"listed","path":"src/models/entities_as_experts.py","file_url":"https://github.com/erolm-a/colla-framework/blob/HEAD/src/models/entities_as_experts.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"8dd59405e934df62"}},{"code_sha256_prefix":"c632e3ec57a98e51","entry":"TokenPred","repo":"erolm-a/colla-framework","repo_kind":"listed","path":"src/models/entities_as_experts.py","file_url":"https://github.com/erolm-a/colla-framework/blob/HEAD/src/models/entities_as_experts.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"c632e3ec57a98e51"}},{"code_sha256_prefix":"002200250a4b148a","entry":"TokenPredHead","repo":"erolm-a/colla-framework","repo_kind":"listed","path":"src/models/entities_as_experts.py","file_url":"https://github.com/erolm-a/colla-framework/blob/HEAD/src/models/entities_as_experts.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"002200250a4b148a"}},{"code_sha256_prefix":"b933236b7948121b","entry":"TruncatedEncoder","repo":"erolm-a/colla-framework","repo_kind":"listed","path":"src/models/entities_as_experts.py","file_url":"https://github.com/erolm-a/colla-framework/blob/HEAD/src/models/entities_as_experts.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"b933236b7948121b"}},{"code_sha256_prefix":"8d4043133ca63339","entry":"TruncatedModel","repo":"erolm-a/colla-framework","repo_kind":"listed","path":"src/models/entities_as_experts.py","file_url":"https://github.com/erolm-a/colla-framework/blob/HEAD/src/models/entities_as_experts.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"8d4043133ca63339"}},{"code_sha256_prefix":"247a61525da9ac3d","entry":"TruncatedModelSecond","repo":"erolm-a/colla-framework","repo_kind":"listed","path":"src/models/entities_as_experts.py","file_url":"https://github.com/erolm-a/colla-framework/blob/HEAD/src/models/entities_as_experts.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"247a61525da9ac3d"}},{"code_sha256_prefix":"a9888b3fd01b6e64","entry":"get_available_device","repo":"erolm-a/colla-framework","repo_kind":"listed","path":"src/models/entities_as_experts.py","file_url":"https://github.com/erolm-a/colla-framework/blob/HEAD/src/models/entities_as_experts.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"a9888b3fd01b6e64"}},{"code_sha256_prefix":"8e19913b399a8cf0","entry":"EntitiesAsExperts","repo":"erolm-a/colla-framework","repo_kind":"listed","path":"src/models/entities_as_experts.py","file_url":"https://github.com/erolm-a/colla-framework/blob/HEAD/src/models/entities_as_experts.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"8e19913b399a8cf0"}},{"code_sha256_prefix":"0590c6e9d6a71ea8","entry":"EntitiesAsExpertsOutputs","repo":"erolm-a/colla-framework","repo_kind":"listed","path":"src/models/entities_as_experts.py","file_url":"https://github.com/erolm-a/colla-framework/blob/HEAD/src/models/entities_as_experts.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"0590c6e9d6a71ea8"}},{"code_sha256_prefix":"9409b589470c9e13","entry":"TorchScriptDumpable","repo":"erolm-a/colla-framework","repo_kind":"listed","path":"src/models/entities_as_experts.py","file_url":"https://github.com/erolm-a/colla-framework/blob/HEAD/src/models/entities_as_experts.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"9409b589470c9e13"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}