{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/elements-of-world-knowledge-ewok-a-cognition","title":"Elements of World Knowledge (EWOK): A cognition-inspired framework for evaluating basic world knowledge in language models","arxiv_id":"2405.09605","date":"2024-05-15","proceeding":null,"authors":["Anna A. Ivanova","Aalok Sathe","Benjamin Lipkin","Unnathi Kumar","Setayesh Radkani","Thomas H. Clark","Carina Kauf","Jennifer Hu","R. T. Pramod","Gabriel Grand","Vivian Paulun","Maria Ryskina","Ekin Akyurek","Ethan Wilcox","Nafisa Rashid","Leshem Chosen","Roger Levy","Evelina Fedorenko","Joshua Tenenbaum","Jacob Andreas"],"abstract":"The ability to build and leverage world models is essential for a general-purpose AI agent. Testing such capabilities is hard, in part because the building blocks of world models are ill-defined. We present Elements of World Knowledge (EWOK), a framework for evaluating world modeling in language models by testing their ability to use knowledge of a concept to match a target text with a plausible/implausible context. EWOK targets specific concepts from multiple knowledge domains known to be vital for world modeling in humans. Domains range from social interactions (help/hinder) to spatial relations (left/right). Both, contexts and targets are minimal pairs. Objects, agents, and locations in the items can be flexibly filled in enabling easy generation of multiple controlled datasets. We then introduce EWOK-CORE-1.0, a dataset of 4,374 items covering 11 world knowledge domains. We evaluate 20 openweights large language models (1.3B--70B parameters) across a battery of evaluation paradigms along with a human norming study comprising 12,480 measurements. The overall performance of all tested models is worse than human performance, with results varying drastically across domains. These data highlight simple cases where even large models fail and present rich avenues for targeted research on LLM world modeling capabilities.","url_abs":"https://arxiv.org/abs/2405.09605v1","url_pdf":"https://arxiv.org/pdf/2405.09605v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"elements-of-world-knowledge-ewok-a-cognition","repo_url":"https://github.com/ewok-core/ewok-paper","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok","spdx":"CC-BY-4.0"}},{"paper_slug":"elements-of-world-knowledge-ewok-a-cognition","repo_url":"https://github.com/ewok-core/ewok","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"none","reach":null}],"tasks":[{"task_slug":"ai-agent","task_name":"AI Agent"},{"task_slug":"world-knowledge","task_name":"World Knowledge"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2405.09605","atlas_url":"https://app.syntology.ai/?focus=2405.09605","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2405.09605"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/ewok-core/ewok","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/ewok-core/ewok-paper","reach":{"status":"ok","spdx":"CC-BY-4.0"}}],"summary":{"ran_draft_wrong":1,"ran":2},"by_repo_kind":{"official":{"samples":3,"ran":3,"repositories":2}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":3,"samples":[{"code_sha256_prefix":"48bb4db3e1669445","entry":"df_to_tuple","repo":"ewok-core/ewok","repo_kind":"official","path":"scripts/run_filtering.py","file_url":"https://github.com/ewok-core/ewok/blob/HEAD/scripts/run_filtering.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"CC-BY-4.0","inline_ok":false,"mcp_get_code":{"code_sha256":"48bb4db3e1669445"}},{"code_sha256_prefix":"22936035ddc71121","entry":"find_fillers","repo":"ewok-core/ewok-paper","repo_kind":"official","path":"ewok/compile/util.py","file_url":"https://github.com/ewok-core/ewok-paper/blob/HEAD/ewok/compile/util.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"CC-BY-4.0","inline_ok":false,"mcp_get_code":{"code_sha256":"22936035ddc71121"}},{"code_sha256_prefix":"811af9f4371dabef","entry":"swap_words","repo":"ewok-core/ewok-paper","repo_kind":"official","path":"ewok/compile/util.py","file_url":"https://github.com/ewok-core/ewok-paper/blob/HEAD/ewok/compile/util.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"CC-BY-4.0","inline_ok":false,"mcp_get_code":{"code_sha256":"811af9f4371dabef"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}