{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/scienceworld-is-your-agent-smarter-than-a-5th","title":"ScienceWorld: Is your Agent Smarter than a 5th Grader?","arxiv_id":"2203.07540","date":"2022-03-14","proceeding":null,"authors":["Ruoyao Wang","Peter Jansen","Marc-Alexandre Côté","Prithviraj Ammanabrolu"],"abstract":"We present ScienceWorld, a benchmark to test agents' scientific reasoning abilities in a new interactive text environment at the level of a standard elementary school science curriculum. Despite the transformer-based progress seen in question-answering and scientific text processing, we find that current models cannot reason about or explain learned science concepts in novel contexts. For instance, models can easily answer what the conductivity of a known material is but struggle when asked how they would conduct an experiment in a grounded environment to find the conductivity of an unknown material. This begs the question of whether current models are simply retrieving answers by way of seeing a large number of similar examples or if they have learned to reason about concepts in a reusable manner. We hypothesize that agents need to be grounded in interactive environments to achieve such reasoning capabilities. Our experiments provide empirical evidence supporting this hypothesis -- showing that a 1.5 million parameter agent trained interactively for 100k steps outperforms a 11 billion parameter model statically trained for scientific question-answering and reasoning from millions of expert demonstrations.","url_abs":"https://arxiv.org/abs/2203.07540v2","url_pdf":"https://arxiv.org/pdf/2203.07540v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"scienceworld-is-your-agent-smarter-than-a-5th","repo_url":"https://github.com/allenai/scienceworld","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"question-answering","task_name":"Question Answering"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2203.07540","atlas_url":"https://app.syntology.ai/?focus=2203.07540","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2203.07540"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/allenai/macaw","reach":{"status":"ok","spdx":"Apache-2.0"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/allenai/scienceworld","reach":{"status":"ok","spdx":"Apache-2.0"}},{"provenance":"deterministic:regex_extraction","url":"https://github.com/allenai/ScienceWorld","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran_draft_wrong":2,"ran":5,"unverified":1},"by_repo_kind":{"official":{"samples":1,"ran":1,"repositories":1},"found_in_text":{"samples":7,"ran":6,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"3c4d106b37c63938","entry":"build_simplification_str","repo":"allenai/scienceworld","repo_kind":"official","path":"examples/random_agent.py","file_url":"https://github.com/allenai/scienceworld/blob/HEAD/examples/random_agent.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"3c4d106b37c63938"}},{"code_sha256_prefix":"20633ae7459cfada","entry":"fix_t5_unk_characters","repo":"allenai/macaw","repo_kind":"found_in_text","path":"macaw/eval_utils.py","file_url":"https://github.com/allenai/macaw/blob/HEAD/macaw/eval_utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"20633ae7459cfada"}},{"code_sha256_prefix":"c30d329a797e15bd","entry":"load_jsonl","repo":"allenai/macaw","repo_kind":"found_in_text","path":"macaw/utils.py","file_url":"https://github.com/allenai/macaw/blob/HEAD/macaw/utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"c30d329a797e15bd"}},{"code_sha256_prefix":"d2166ef001a9e416","entry":"replace_punctuation","repo":"allenai/macaw","repo_kind":"found_in_text","path":"macaw/eval_utils.py","file_url":"https://github.com/allenai/macaw/blob/HEAD/macaw/eval_utils.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"d2166ef001a9e416"}},{"code_sha256_prefix":"0ff0cabbc0770012","entry":"run_model","repo":"allenai/macaw","repo_kind":"found_in_text","path":"macaw/utils.py","file_url":"https://github.com/allenai/macaw/blob/HEAD/macaw/utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"0ff0cabbc0770012"}},{"code_sha256_prefix":"df88f11724e3cab5","entry":"run_model_with_outputs","repo":"allenai/macaw","repo_kind":"found_in_text","path":"macaw/batch_eval.py","file_url":"https://github.com/allenai/macaw/blob/HEAD/macaw/batch_eval.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"df88f11724e3cab5"}},{"code_sha256_prefix":"00139853f0c3a89e","entry":"score_string_similarity","repo":"allenai/macaw","repo_kind":"found_in_text","path":"macaw/eval_utils.py","file_url":"https://github.com/allenai/macaw/blob/HEAD/macaw/eval_utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"00139853f0c3a89e"}},{"code_sha256_prefix":"a96d409802f5ecdc","entry":"load_model","repo":"allenai/macaw","repo_kind":"found_in_text","path":"macaw/utils.py","file_url":"https://github.com/allenai/macaw/blob/HEAD/macaw/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"a96d409802f5ecdc"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}