{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/hcast-human-calibrated-autonomy-software","title":"HCAST: Human-Calibrated Autonomy Software Tasks","arxiv_id":"2503.17354","date":"2025-03-21","proceeding":null,"authors":["David Rein","Joel Becker","Amy Deng","Seraphina Nix","Chris Canal","Daniel O'Connel","Pip Arnott","Ryan Bloom","Thomas Broadley","Katharyn Garcia","Brian Goodrich","Max Hasin","Sami Jawhar","Megan Kinniment","Thomas Kwa","Aron Lajko","Nate Rush","Lucas Jun Koba Sato","Sydney von Arx","Ben West","Lawrence Chan","Elizabeth Barnes"],"abstract":"To understand and predict the societal impacts of highly autonomous AI systems, we need benchmarks with grounding, i.e., metrics that directly connect AI performance to real-world effects we care about. We present HCAST (Human-Calibrated Autonomy Software Tasks), a benchmark of 189 machine learning engineering, cybersecurity, software engineering, and general reasoning tasks. We collect 563 human baselines (totaling over 1500 hours) from people skilled in these domains, working under identical conditions as AI agents, which lets us estimate that HCAST tasks take humans between one minute and 8+ hours. Measuring the time tasks take for humans provides an intuitive metric for evaluating AI capabilities, helping answer the question \"can an agent be trusted to complete a task that would take a human X hours?\" We evaluate the success rates of AI agents built on frontier foundation models, and we find that current agents succeed 70-80% of the time on tasks that take humans less than one hour, and less than 20% of the time on tasks that take humans more than 4 hours.","url_abs":"https://arxiv.org/abs/2503.17354v1","url_pdf":"https://arxiv.org/pdf/2503.17354v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"hcast-human-calibrated-autonomy-software","repo_url":"https://github.com/METR/hcast-public","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"none","reach":null}],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2503.17354","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2503.17354"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/METR/hcast-public","reach":null}],"summary":{"unverified":3},"by_repo_kind":{"official":{"samples":3,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":3,"samples":[{"code_sha256_prefix":"26d81e9f087b8303","entry":"check_raw_hypotheses_hash","repo":"METR/hcast-public","repo_kind":"official","path":"hypothesis_testing/hypotheses.py","file_url":"https://github.com/METR/hcast-public/blob/HEAD/hypothesis_testing/hypotheses.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"26d81e9f087b8303"}},{"code_sha256_prefix":"1d3685b9629b1825","entry":"generate_list","repo":"METR/hcast-public","repo_kind":"official","path":"hypothesis_testing/hypotheses.py","file_url":"https://github.com/METR/hcast-public/blob/HEAD/hypothesis_testing/hypotheses.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"1d3685b9629b1825"}},{"code_sha256_prefix":"16906be22e3f7dee","entry":"get_correct","repo":"METR/hcast-public","repo_kind":"official","path":"hypothesis_testing/hypotheses.py","file_url":"https://github.com/METR/hcast-public/blob/HEAD/hypothesis_testing/hypotheses.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"16906be22e3f7dee"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}