{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/worldsense-a-synthetic-benchmark-for-grounded","title":"WorldSense: A Synthetic Benchmark for Grounded Reasoning in Large Language Models","arxiv_id":"2311.15930","date":"2023-11-27","proceeding":null,"authors":["Youssef Benchekroun","Megi Dervishi","Mark Ibrahim","Jean-Baptiste Gaya","Xavier Martinet","Grégoire Mialon","Thomas Scialom","Emmanuel Dupoux","Dieuwke Hupkes","Pascal Vincent"],"abstract":"We propose WorldSense, a benchmark designed to assess the extent to which LLMs are consistently able to sustain tacit world models, by testing how they draw simple inferences from descriptions of simple arrangements of entities. Worldsense is a synthetic benchmark with three problem types, each with their own trivial control, which explicitly avoids bias by decorrelating the abstract structure of problems from the vocabulary and expressions, and by decorrelating all problem subparts with the correct response. We run our benchmark on three state-of-the-art chat-LLMs (GPT3.5, GPT4 and Llama2-chat) and show that these models make errors even with as few as three objects. Furthermore, they have quite heavy response biases, preferring certain responses irrespective of the question. Errors persist even with chain-of-thought prompting and in-context learning. Lastly, we show that while finetuning on similar problems does result in substantial improvements -- within- and out-of-distribution -- the finetuned models do not generalise beyond a constraint problem space.","url_abs":"https://arxiv.org/abs/2311.15930v1","url_pdf":"https://arxiv.org/pdf/2311.15930v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"worldsense-a-synthetic-benchmark-for-grounded","repo_url":"https://github.com/facebookresearch/worldsense","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"none","reach":null}],"tasks":[{"task_slug":"in-context-learning","task_name":"In-Context Learning"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2311.15930","atlas_url":"https://app.syntology.ai/?focus=2311.15930","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2311.15930"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/facebookresearch/worldsense","reach":null}],"summary":{"ran_draft_wrong":2},"by_repo_kind":{"official":{"samples":2,"ran":2,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":2,"samples":[{"code_sha256_prefix":"810abced304c4a19","entry":"load_testset","repo":"facebookresearch/worldsense","repo_kind":"official","path":"worldsense/benchmark.py","file_url":"https://github.com/facebookresearch/worldsense/blob/HEAD/worldsense/benchmark.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"810abced304c4a19"}},{"code_sha256_prefix":"599f56b6edd3e35f","entry":"load_trainset","repo":"facebookresearch/worldsense","repo_kind":"official","path":"worldsense/benchmark.py","file_url":"https://github.com/facebookresearch/worldsense/blob/HEAD/worldsense/benchmark.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"599f56b6edd3e35f"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}