{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/turtlebench-evaluating-top-language-models","title":"TurtleBench: Evaluating Top Language Models via Real-World Yes/No Puzzles","arxiv_id":"2410.05262","date":"2024-10-07","proceeding":null,"authors":["Qingchen Yu","Shichao Song","Ke Fang","Yunfeng Shi","Zifan Zheng","Hanyu Wang","Simin Niu","Zhiyu Li"],"abstract":"As the application of Large Language Models (LLMs) expands, the demand for reliable evaluations increases. Existing LLM evaluation benchmarks primarily rely on static datasets, making it challenging to assess model performance in dynamic interactions with users. Moreover, these benchmarks often depend on specific background knowledge, complicating the measurement of a model's logical reasoning capabilities. Other dynamic evaluation methods based on strong models or manual efforts may introduce biases and incur high costs and time demands, hindering large-scale application. To address these issues, we propose TurtleBench. TurtleBench collects real user guesses from our online Turtle Soup Puzzle platform that we developed. This approach allows for the relatively dynamic generation of evaluation datasets, mitigating the risk of model cheating while aligning assessments more closely with genuine user needs for reasoning capabilities, thus enhancing the reliability of evaluations. TurtleBench includes 1,532 user guesses along with the correctness of guesses after annotation. Using this dataset, we thoroughly evaluated nine of the most advanced LLMs available today. Notably, the OpenAI o1 series models did not achieve leading results in these evaluations. We propose several hypotheses for further research, such as \"the latent reasoning of o1 utilizes trivial Chain-of-Thought (CoT) techniques\" and \"increasing CoT length not only provides reasoning benefits but also incurs noise costs.\"","url_abs":"https://arxiv.org/abs/2410.05262v1","url_pdf":"https://arxiv.org/pdf/2410.05262v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"turtlebench-evaluating-top-language-models","repo_url":"https://github.com/mazzzystar/TurtleBench","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"logical-reasoning","task_name":"Logical Reasoning"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2410.05262","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2410.05262"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/mazzzystar/TurtleBench","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran":3,"unverified":2},"by_repo_kind":{"official":{"samples":5,"ran":3,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"1f41ba18b8753d1f","entry":"get_token_usage","repo":"mazzzystar/TurtleBench","repo_kind":"official","path":"insights/token_cal.py","file_url":"https://github.com/mazzzystar/TurtleBench/blob/HEAD/insights/token_cal.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"1f41ba18b8753d1f"}},{"code_sha256_prefix":"5fa76cf6d46c6050","entry":"is_o1_model","repo":"mazzzystar/TurtleBench","repo_kind":"official","path":"models.py","file_url":"https://github.com/mazzzystar/TurtleBench/blob/HEAD/models.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"5fa76cf6d46c6050"}},{"code_sha256_prefix":"eda1e0ba450c8b0f","entry":"load_latest_logs","repo":"mazzzystar/TurtleBench","repo_kind":"official","path":"analyst.py","file_url":"https://github.com/mazzzystar/TurtleBench/blob/HEAD/analyst.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"eda1e0ba450c8b0f"}},{"code_sha256_prefix":"b6a420ef1adb8bef","entry":"call_api","repo":"mazzzystar/TurtleBench","repo_kind":"official","path":"models.py","file_url":"https://github.com/mazzzystar/TurtleBench/blob/HEAD/models.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"b6a420ef1adb8bef"}},{"code_sha256_prefix":"66eef613262a8a05","entry":"get_model_config","repo":"mazzzystar/TurtleBench","repo_kind":"official","path":"models.py","file_url":"https://github.com/mazzzystar/TurtleBench/blob/HEAD/models.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"66eef613262a8a05"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}