{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/chatlog-recording-and-analyzing-chatgpt","title":"ChatLog: Carefully Evaluating the Evolution of ChatGPT Across Time","arxiv_id":"2304.14106","date":"2023-04-27","proceeding":null,"authors":["Shangqing Tu","Chunyang Li","Jifan Yu","Xiaozhi Wang","Lei Hou","Juanzi Li"],"abstract":"ChatGPT has achieved great success and can be considered to have acquired an infrastructural status. There are abundant works for evaluating ChatGPT on benchmarks. However, existing benchmarks encounter two challenges: (1) Disregard for periodical evaluation and (2) Lack of fine-grained features. In this paper, we construct ChatLog, an ever-updating dataset with large-scale records of diverse long-form ChatGPT responses for 21 NLP benchmarks from March, 2023 to now. We conduct a comprehensive performance evaluation to find that most capabilities of ChatGPT improve over time except for some abilities, and there exists a step-wise evolving pattern of ChatGPT. We further analyze the inherent characteristics of ChatGPT by extracting the knowledge and linguistic features. We find some stable features that stay unchanged and apply them on the detection of ChatGPT-generated texts to improve the robustness of cross-version detection. We will continuously maintain our project at \\url{https://github.com/THU-KEG/ChatLog/}.","url_abs":"https://arxiv.org/abs/2304.14106v2","url_pdf":"https://arxiv.org/pdf/2304.14106v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"chatlog-recording-and-analyzing-chatgpt","repo_url":"https://github.com/thu-keg/chatlog","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"natural-language-understanding","task_name":"Natural Language Understanding"}],"methods":[],"datasets_introduced":[{"slug":"chatlog","name":"ChatLog","full_name":""}],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2304.14106","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2304.14106"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/thu-keg/chatlog","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"unverified":2},"by_repo_kind":{"official":{"samples":2,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"d53fe07debf38d2b","entry":"calc_avg_var_cov","repo":"thu-keg/chatlog","repo_kind":"official","path":"model/diff.py","file_url":"https://github.com/thu-keg/chatlog/blob/HEAD/model/diff.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d53fe07debf38d2b"}},{"code_sha256_prefix":"a76ceca7044b5499","entry":"temporal_correlation_coefficient","repo":"thu-keg/chatlog","repo_kind":"official","path":"model/visualization.py","file_url":"https://github.com/thu-keg/chatlog/blob/HEAD/model/visualization.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a76ceca7044b5499"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}