{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/workbench-a-benchmark-dataset-for-agents-in-a","title":"WorkBench: a Benchmark Dataset for Agents in a Realistic Workplace Setting","arxiv_id":"2405.00823","date":"2024-05-01","proceeding":null,"authors":["Olly Styles","Sam Miller","Patricio Cerda-Mardini","Tanaya Guha","Victor Sanchez","Bertie Vidgen"],"abstract":"We introduce WorkBench: a benchmark dataset for evaluating agents' ability to execute tasks in a workplace setting. WorkBench contains a sandbox environment with five databases, 26 tools, and 690 tasks. These tasks represent common business activities, such as sending emails and scheduling meetings. The tasks in WorkBench are challenging as they require planning, tool selection, and often multiple actions. If a task has been successfully executed, one (or more) of the database values may change. The correct outcome for each task is unique and unambiguous, which allows for robust, automated evaluation. We call this key contribution outcome-centric evaluation. We evaluate five existing ReAct agents on WorkBench, finding they successfully complete as few as 3% of tasks (Llama2-70B), and just 43% for the best-performing (GPT-4). We further find that agents' errors can result in the wrong action being taken, such as an email being sent to the wrong person. WorkBench reveals weaknesses in agents' ability to undertake common business activities, raising questions about their use in high-stakes workplace settings. WorkBench is publicly available as a free resource at https://github.com/olly-styles/WorkBench.","url_abs":"https://arxiv.org/abs/2405.00823v2","url_pdf":"https://arxiv.org/pdf/2405.00823v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"workbench-a-benchmark-dataset-for-agents-in-a","repo_url":"https://github.com/olly-styles/workbench","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"scheduling","task_name":"Scheduling"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2405.00823","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2405.00823"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/olly-styles/workbench","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":12,"unverified":1},"by_repo_kind":{"official":{"samples":13,"ran":12,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"0a9a03a51369c346","entry":"accept_either_chart_end_date","repo":"olly-styles/workbench","repo_kind":"official","path":"src/evals/evaluation.py","file_url":"https://github.com/olly-styles/workbench/blob/HEAD/src/evals/evaluation.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"0a9a03a51369c346"}},{"code_sha256_prefix":"82d39fede4039848","entry":"convert_intermediate_step_to_function_call","repo":"olly-styles/workbench","repo_kind":"official","path":"src/evals/actions.py","file_url":"https://github.com/olly-styles/workbench/blob/HEAD/src/evals/actions.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"82d39fede4039848"}},{"code_sha256_prefix":"d9798b807ed518df","entry":"end_date_minor_error","repo":"olly-styles/workbench","repo_kind":"official","path":"src/evals/evaluation.py","file_url":"https://github.com/olly-styles/workbench/blob/HEAD/src/evals/evaluation.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d9798b807ed518df"}},{"code_sha256_prefix":"dea9fe8105e2dc28","entry":"get_first_free_slot","repo":"olly-styles/workbench","repo_kind":"official","path":"src/data_generation/data_generation_utils.py","file_url":"https://github.com/olly-styles/workbench/blob/HEAD/src/data_generation/data_generation_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"dea9fe8105e2dc28"}},{"code_sha256_prefix":"26778cb3a7db5e73","entry":"get_function_name","repo":"olly-styles/workbench","repo_kind":"official","path":"src/evals/evaluation.py","file_url":"https://github.com/olly-styles/workbench/blob/HEAD/src/evals/evaluation.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"26778cb3a7db5e73"}},{"code_sha256_prefix":"555b4c7cfe86c224","entry":"get_random_future_date","repo":"olly-styles/workbench","repo_kind":"official","path":"src/data_generation/data_generation_utils.py","file_url":"https://github.com/olly-styles/workbench/blob/HEAD/src/data_generation/data_generation_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"555b4c7cfe86c224"}},{"code_sha256_prefix":"7c6964aeaeaaa3e5","entry":"get_random_future_datetime","repo":"olly-styles/workbench","repo_kind":"official","path":"src/data_generation/data_generation_utils.py","file_url":"https://github.com/olly-styles/workbench/blob/HEAD/src/data_generation/data_generation_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"7c6964aeaeaaa3e5"}},{"code_sha256_prefix":"47b624708c307a0c","entry":"ground_truth_version_for_results","repo":"olly-styles/workbench","repo_kind":"official","path":"src/evals/metrics.py","file_url":"https://github.com/olly-styles/workbench/blob/HEAD/src/evals/metrics.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"47b624708c307a0c"}},{"code_sha256_prefix":"eb3f0fbaf1a421dc","entry":"meta_path_for_results","repo":"olly-styles/workbench","repo_kind":"official","path":"src/evals/metrics.py","file_url":"https://github.com/olly-styles/workbench/blob/HEAD/src/evals/metrics.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"eb3f0fbaf1a421dc"}},{"code_sha256_prefix":"1ef3e6bd912b699a","entry":"money","repo":"olly-styles/workbench","repo_kind":"official","path":"retro/figs/plot_cost_vs_accuracy.py","file_url":"https://github.com/olly-styles/workbench/blob/HEAD/retro/figs/plot_cost_vs_accuracy.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"1ef3e6bd912b699a"}},{"code_sha256_prefix":"a3db7ab2f37601d6","entry":"parse_action","repo":"olly-styles/workbench","repo_kind":"official","path":"src/evals/agent.py","file_url":"https://github.com/olly-styles/workbench/blob/HEAD/src/evals/agent.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a3db7ab2f37601d6"}},{"code_sha256_prefix":"6871aad32f38dd25","entry":"strip_results_suffix","repo":"olly-styles/workbench","repo_kind":"official","path":"src/evals/metrics.py","file_url":"https://github.com/olly-styles/workbench/blob/HEAD/src/evals/metrics.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"6871aad32f38dd25"}},{"code_sha256_prefix":"c12489c977a3c127","entry":"resolve_route","repo":"olly-styles/workbench","repo_kind":"official","path":"src/evals/agent.py","file_url":"https://github.com/olly-styles/workbench/blob/HEAD/src/evals/agent.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c12489c977a3c127"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}