{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/gpt-fathom-benchmarking-large-language-models","title":"GPT-Fathom: Benchmarking Large Language Models to Decipher the Evolutionary Path towards GPT-4 and Beyond","arxiv_id":"2309.16583","date":"2023-09-28","proceeding":null,"authors":["Shen Zheng","Yuyu Zhang","Yijie Zhu","Chenguang Xi","Pengyang Gao","Xun Zhou","Kevin Chen-Chuan Chang"],"abstract":"With the rapid advancement of large language models (LLMs), there is a pressing need for a comprehensive evaluation suite to assess their capabilities and limitations. Existing LLM leaderboards often reference scores reported in other papers without consistent settings and prompts, which may inadvertently encourage cherry-picking favored settings and prompts for better results. In this work, we introduce GPT-Fathom, an open-source and reproducible LLM evaluation suite built on top of OpenAI Evals. We systematically evaluate 10+ leading LLMs as well as OpenAI's legacy models on 20+ curated benchmarks across 7 capability categories, all under aligned settings. Our retrospective study on OpenAI's earlier models offers valuable insights into the evolutionary path from GPT-3 to GPT-4. Currently, the community is eager to know how GPT-3 progressively improves to GPT-4, including technical details like whether adding code data improves LLM's reasoning capability, which aspects of LLM capability can be improved by SFT and RLHF, how much is the alignment tax, etc. Our analysis sheds light on many of these questions, aiming to improve the transparency of advanced LLMs.","url_abs":"https://arxiv.org/abs/2309.16583v6","url_pdf":"https://arxiv.org/pdf/2309.16583v6.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"gpt-fathom-benchmarking-large-language-models","repo_url":"https://github.com/gpt-fathom/gpt-fathom","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"benchmarking","task_name":"Benchmarking"}],"methods":[{"method_slug":"absolute-position-encodings","method_name":"Absolute Position Encodings"},{"method_slug":"adam","method_name":"Adam"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"attention-dropout","method_name":"Attention Dropout"},{"method_slug":"bpe","method_name":"BPE"},{"method_slug":"cosine-annealing","method_name":"Cosine Annealing"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"gpt-3","method_name":"GPT-3"},{"method_slug":"gpt-4","method_name":"GPT-4"},{"method_slug":"label-smoothing","method_name":"Label Smoothing"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"linear-warmup-with-cosine-annealing","method_name":"Linear Warmup With Cosine Annealing"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"position-wise-feed-forward-layer","method_name":"Position-Wise Feed-Forward Layer"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"sft","method_name":"SFT"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"transformer","method_name":"Transformer"},{"method_slug":"weight-decay","method_name":"Weight Decay"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2309.16583","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2309.16583"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/gpt-fathom/gpt-fathom","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":5,"unverified":4},"by_repo_kind":{"official":{"samples":9,"ran":5,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"0e23050b77c7e881","entry":"create_chat_prompt","repo":"gpt-fathom/gpt-fathom","repo_kind":"official","path":"evals/build_data/build_arc.py","file_url":"https://github.com/gpt-fathom/gpt-fathom/blob/HEAD/evals/build_data/build_arc.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"0e23050b77c7e881"}},{"code_sha256_prefix":"43cd489d780c0a6a","entry":"create_chat_prompt","repo":"gpt-fathom/gpt-fathom","repo_kind":"official","path":"evals/build_data/build_bbh.py","file_url":"https://github.com/gpt-fathom/gpt-fathom/blob/HEAD/evals/build_data/build_bbh.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"43cd489d780c0a6a"}},{"code_sha256_prefix":"7d527676673e7020","entry":"create_fewshot_prompt","repo":"gpt-fathom/gpt-fathom","repo_kind":"official","path":"evals/build_data/build_arc.py","file_url":"https://github.com/gpt-fathom/gpt-fathom/blob/HEAD/evals/build_data/build_arc.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"7d527676673e7020"}},{"code_sha256_prefix":"4722bc8f6adb2e0b","entry":"make_abc","repo":"gpt-fathom/gpt-fathom","repo_kind":"official","path":"evals/formatting.py","file_url":"https://github.com/gpt-fathom/gpt-fathom/blob/HEAD/evals/formatting.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"4722bc8f6adb2e0b"}},{"code_sha256_prefix":"b320a96ace235fa2","entry":"n_ctx_from_model_name","repo":"gpt-fathom/gpt-fathom","repo_kind":"official","path":"evals/registry.py","file_url":"https://github.com/gpt-fathom/gpt-fathom/blob/HEAD/evals/registry.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b320a96ace235fa2"}},{"code_sha256_prefix":"d7e42b6245f2db86","entry":"gzip_open","repo":"gpt-fathom/gpt-fathom","repo_kind":"official","path":"evals/data.py","file_url":"https://github.com/gpt-fathom/gpt-fathom/blob/HEAD/evals/data.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d7e42b6245f2db86"}},{"code_sha256_prefix":"a93524060abbb7bc","entry":"record_embedding","repo":"gpt-fathom/gpt-fathom","repo_kind":"official","path":"evals/record.py","file_url":"https://github.com/gpt-fathom/gpt-fathom/blob/HEAD/evals/record.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a93524060abbb7bc"}},{"code_sha256_prefix":"de0de0f8ca23f85e","entry":"record_match","repo":"gpt-fathom/gpt-fathom","repo_kind":"official","path":"evals/record.py","file_url":"https://github.com/gpt-fathom/gpt-fathom/blob/HEAD/evals/record.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"de0de0f8ca23f85e"}},{"code_sha256_prefix":"6c1640a35af7cdb9","entry":"record_sampling","repo":"gpt-fathom/gpt-fathom","repo_kind":"official","path":"evals/record.py","file_url":"https://github.com/gpt-fathom/gpt-fathom/blob/HEAD/evals/record.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"6c1640a35af7cdb9"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}