{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/gpqa-a-graduate-level-google-proof-q-a","title":"GPQA: A Graduate-Level Google-Proof Q&A Benchmark","arxiv_id":"2311.12022","date":"2023-11-20","proceeding":null,"authors":["David Rein","Betty Li Hou","Asa Cooper Stickland","Jackson Petty","Richard Yuanzhe Pang","Julien Dirani","Julian Michael","Samuel R. Bowman"],"abstract":"We present GPQA, a challenging dataset of 448 multiple-choice questions written by domain experts in biology, physics, and chemistry. We ensure that the questions are high-quality and extremely difficult: experts who have or are pursuing PhDs in the corresponding domains reach 65% accuracy (74% when discounting clear mistakes the experts identified in retrospect), while highly skilled non-expert validators only reach 34% accuracy, despite spending on average over 30 minutes with unrestricted access to the web (i.e., the questions are \"Google-proof\"). The questions are also difficult for state-of-the-art AI systems, with our strongest GPT-4 based baseline achieving 39% accuracy. If we are to use future AI systems to help us answer very hard questions, for example, when developing new scientific knowledge, we need to develop scalable oversight methods that enable humans to supervise their outputs, which may be difficult even if the supervisors are themselves skilled and knowledgeable. The difficulty of GPQA both for skilled non-experts and frontier AI systems should enable realistic scalable oversight experiments, which we hope can help devise ways for human experts to reliably get truthful information from AI systems that surpass human capabilities.","url_abs":"https://arxiv.org/abs/2311.12022v1","url_pdf":"https://arxiv.org/pdf/2311.12022v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"gpqa-a-graduate-level-google-proof-q-a","repo_url":"https://github.com/idavidrein/gpqa","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"gpqa-a-graduate-level-google-proof-q-a","repo_url":"https://github.com/tml-epfl/os-harm","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":null},{"paper_slug":"gpqa-a-graduate-level-google-proof-q-a","repo_url":"https://github.com/RUCAIBox/LLMBox","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"none","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"multiple-choice","task_name":"Multiple-choice"}],"methods":[{"method_slug":"absolute-position-encodings","method_name":"Absolute Position Encodings"},{"method_slug":"adam","method_name":"Adam"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"bpe","method_name":"BPE"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"gpt-4","method_name":"GPT-4"},{"method_slug":"label-smoothing","method_name":"Label Smoothing"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"position-wise-feed-forward-layer","method_name":"Position-Wise Feed-Forward Layer"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"transformer","method_name":"Transformer"}],"datasets_introduced":[{"slug":"gpqa","name":"GPQA","full_name":""}],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2311.12022","atlas_url":"https://app.syntology.ai/?focus=2311.12022","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2311.12022"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/tml-epfl/os-harm","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/RUCAIBox/LLMBox","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/idavidrein/gpqa","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":7,"unverified":1},"by_repo_kind":{"official":{"samples":8,"ran":7,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"bf93745bda87d8d6","entry":"f1_score","repo":"idavidrein/gpqa","repo_kind":"official","path":"baselines/open_book.py","file_url":"https://github.com/idavidrein/gpqa/blob/HEAD/baselines/open_book.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"bf93745bda87d8d6"}},{"code_sha256_prefix":"509cf2260198920d","entry":"generate_prompt_from_examples","repo":"idavidrein/gpqa","repo_kind":"official","path":"baselines/utils.py","file_url":"https://github.com/idavidrein/gpqa/blob/HEAD/baselines/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"509cf2260198920d"}},{"code_sha256_prefix":"c2c45901f9627f5e","entry":"generate_question_and_answers","repo":"idavidrein/gpqa","repo_kind":"official","path":"baselines/answer_cbow.py","file_url":"https://github.com/idavidrein/gpqa/blob/HEAD/baselines/answer_cbow.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c2c45901f9627f5e"}},{"code_sha256_prefix":"ac2b95c8fa3bc645","entry":"generate_question_and_answers","repo":"idavidrein/gpqa","repo_kind":"official","path":"baselines/answer_only.py","file_url":"https://github.com/idavidrein/gpqa/blob/HEAD/baselines/answer_only.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ac2b95c8fa3bc645"}},{"code_sha256_prefix":"a5691fda5dd24a86","entry":"load_from_json","repo":"idavidrein/gpqa","repo_kind":"official","path":"baselines/utils.py","file_url":"https://github.com/idavidrein/gpqa/blob/HEAD/baselines/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a5691fda5dd24a86"}},{"code_sha256_prefix":"bc5679c991d7d2e6","entry":"remove_punctuation","repo":"idavidrein/gpqa","repo_kind":"official","path":"baselines/open_book.py","file_url":"https://github.com/idavidrein/gpqa/blob/HEAD/baselines/open_book.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"bc5679c991d7d2e6"}},{"code_sha256_prefix":"38400f169c72be8d","entry":"tokenize_data","repo":"idavidrein/gpqa","repo_kind":"official","path":"baselines/answer_cbow.py","file_url":"https://github.com/idavidrein/gpqa/blob/HEAD/baselines/answer_cbow.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"38400f169c72be8d"}},{"code_sha256_prefix":"008fc7c26a7804df","entry":"add_position_data","repo":"idavidrein/gpqa","repo_kind":"official","path":"baselines/answer_only.py","file_url":"https://github.com/idavidrein/gpqa/blob/HEAD/baselines/answer_only.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"008fc7c26a7804df"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}