{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/how-close-is-chatgpt-to-human-experts","title":"How Close is ChatGPT to Human Experts? Comparison Corpus, Evaluation, and Detection","arxiv_id":"2301.07597","date":"2023-01-18","proceeding":null,"authors":["Biyang Guo","Xin Zhang","Ziyuan Wang","Minqi Jiang","Jinran Nie","Yuxuan Ding","Jianwei Yue","Yupeng Wu"],"abstract":"The introduction of ChatGPT has garnered widespread attention in both academic and industrial communities. ChatGPT is able to respond effectively to a wide range of human questions, providing fluent and comprehensive answers that significantly surpass previous public chatbots in terms of security and usefulness. On one hand, people are curious about how ChatGPT is able to achieve such strength and how far it is from human experts. On the other hand, people are starting to worry about the potential negative impacts that large language models (LLMs) like ChatGPT could have on society, such as fake news, plagiarism, and social security issues. In this work, we collected tens of thousands of comparison responses from both human experts and ChatGPT, with questions ranging from open-domain, financial, medical, legal, and psychological areas. We call the collected dataset the Human ChatGPT Comparison Corpus (HC3). Based on the HC3 dataset, we study the characteristics of ChatGPT's responses, the differences and gaps from human experts, and future directions for LLMs. We conducted comprehensive human evaluations and linguistic analyses of ChatGPT-generated content compared with that of humans, where many interesting results are revealed. After that, we conduct extensive experiments on how to effectively detect whether a certain text is generated by ChatGPT or humans. We build three different detection systems, explore several key factors that influence their effectiveness, and evaluate them in different scenarios. The dataset, code, and models are all publicly available at https://github.com/Hello-SimpleAI/chatgpt-comparison-detection.","url_abs":"https://arxiv.org/abs/2301.07597v1","url_pdf":"https://arxiv.org/pdf/2301.07597v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"how-close-is-chatgpt-to-human-experts","repo_url":"https://github.com/hello-simpleai/chatgpt-comparison-detection","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}},{"paper_slug":"how-close-is-chatgpt-to-human-experts","repo_url":"https://github.com/loris3/evaluation_explanation_quality","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"how-close-is-chatgpt-to-human-experts","repo_url":"https://github.com/lsc-1/pecola","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2301.07597","atlas_url":"https://app.syntology.ai/?focus=2301.07597","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2301.07597"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/loris3/evaluation_explanation_quality","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/hello-simpleai/chatgpt-comparison-detection","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/lsc-1/pecola","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":11,"unverified":14},"by_repo_kind":{"listed":{"samples":25,"ran":11,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"61f9329047b0ab17","entry":"get_base_score","repo":"lsc-1/pecola","repo_kind":"listed","path":"lib/inference/base.py","file_url":"https://github.com/lsc-1/pecola/blob/HEAD/lib/inference/base.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"61f9329047b0ab17"}},{"code_sha256_prefix":"29588baf96ee353d","entry":"get_dropout_score","repo":"lsc-1/pecola","repo_kind":"listed","path":"lib/inference/dropout.py","file_url":"https://github.com/lsc-1/pecola/blob/HEAD/lib/inference/dropout.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"29588baf96ee353d"}},{"code_sha256_prefix":"b78ad0b30fc6a1da","entry":"get_model1","repo":"lsc-1/pecola","repo_kind":"listed","path":"lib/networks.py","file_url":"https://github.com/lsc-1/pecola/blob/HEAD/lib/networks.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b78ad0b30fc6a1da"}},{"code_sha256_prefix":"a52d92262c816283","entry":"get_msp_score","repo":"lsc-1/pecola","repo_kind":"listed","path":"lib/inference/dropout.py","file_url":"https://github.com/lsc-1/pecola/blob/HEAD/lib/inference/dropout.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a52d92262c816283"}},{"code_sha256_prefix":"41058bada6b6f29d","entry":"get_num_labels","repo":"lsc-1/pecola","repo_kind":"listed","path":"lib/exp.py","file_url":"https://github.com/lsc-1/pecola/blob/HEAD/lib/exp.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"41058bada6b6f29d"}},{"code_sha256_prefix":"6c82794bd0139e76","entry":"marginLoss","repo":"lsc-1/pecola","repo_kind":"listed","path":"lib/common.py","file_url":"https://github.com/lsc-1/pecola/blob/HEAD/lib/common.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"6c82794bd0139e76"}},{"code_sha256_prefix":"e97eacfdcbaf68a1","entry":"sclLoss","repo":"lsc-1/pecola","repo_kind":"listed","path":"lib/common.py","file_url":"https://github.com/lsc-1/pecola/blob/HEAD/lib/common.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e97eacfdcbaf68a1"}},{"code_sha256_prefix":"96ab70351169ff04","entry":"set_device_and_seed","repo":"lsc-1/pecola","repo_kind":"listed","path":"select_strategy.py","file_url":"https://github.com/lsc-1/pecola/blob/HEAD/select_strategy.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"96ab70351169ff04"}},{"code_sha256_prefix":"6a16ec67a0221341","entry":"set_device_and_seed","repo":"lsc-1/pecola","repo_kind":"listed","path":"augumentation_data/yake_mask.py","file_url":"https://github.com/lsc-1/pecola/blob/HEAD/augumentation_data/yake_mask.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"6a16ec67a0221341"}},{"code_sha256_prefix":"be4d25103faa09a4","entry":"small_fix","repo":"lsc-1/pecola","repo_kind":"listed","path":"augumentation_data/mask_filling.py","file_url":"https://github.com/lsc-1/pecola/blob/HEAD/augumentation_data/mask_filling.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"be4d25103faa09a4"}},{"code_sha256_prefix":"50ec38fc4fd612e1","entry":"test_acc","repo":"lsc-1/pecola","repo_kind":"listed","path":"lib/common.py","file_url":"https://github.com/lsc-1/pecola/blob/HEAD/lib/common.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"50ec38fc4fd612e1"}},{"code_sha256_prefix":"a6359426276fb40b","entry":"auroc","repo":"lsc-1/pecola","repo_kind":"listed","path":"lib/metrics.py","file_url":"https://github.com/lsc-1/pecola/blob/HEAD/lib/metrics.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a6359426276fb40b"}},{"code_sha256_prefix":"d1c6ff9d27f3294b","entry":"get_FFN_features","repo":"lsc-1/pecola","repo_kind":"listed","path":"lib/inference/draw.py","file_url":"https://github.com/lsc-1/pecola/blob/HEAD/lib/inference/draw.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d1c6ff9d27f3294b"}},{"code_sha256_prefix":"0256d90810fbc438","entry":"get_cse_features","repo":"lsc-1/pecola","repo_kind":"listed","path":"lib/inference/draw.py","file_url":"https://github.com/lsc-1/pecola/blob/HEAD/lib/inference/draw.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"0256d90810fbc438"}},{"code_sha256_prefix":"13081870f2ef7a7d","entry":"get_d2u_score","repo":"lsc-1/pecola","repo_kind":"listed","path":"lib/inference/base.py","file_url":"https://github.com/lsc-1/pecola/blob/HEAD/lib/inference/base.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"13081870f2ef7a7d"}},{"code_sha256_prefix":"dd94c31be0cd6720","entry":"get_features","repo":"lsc-1/pecola","repo_kind":"listed","path":"lib/inference/draw.py","file_url":"https://github.com/lsc-1/pecola/blob/HEAD/lib/inference/draw.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"dd94c31be0cd6720"}},{"code_sha256_prefix":"e614043206c6e7fd","entry":"get_grad_norm_score","repo":"lsc-1/pecola","repo_kind":"listed","path":"lib/inference/base.py","file_url":"https://github.com/lsc-1/pecola/blob/HEAD/lib/inference/base.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e614043206c6e7fd"}},{"code_sha256_prefix":"7b9990d7d4520898","entry":"get_is_pos","repo":"lsc-1/pecola","repo_kind":"listed","path":"lib/metrics.py","file_url":"https://github.com/lsc-1/pecola/blob/HEAD/lib/metrics.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"7b9990d7d4520898"}},{"code_sha256_prefix":"9aab08fc71951fbb","entry":"get_key_positions","repo":"lsc-1/pecola","repo_kind":"listed","path":"lib/inference/generation.py","file_url":"https://github.com/lsc-1/pecola/blob/HEAD/lib/inference/generation.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"9aab08fc71951fbb"}},{"code_sha256_prefix":"24c1d85c8d245fbb","entry":"get_keyword_confidence","repo":"lsc-1/pecola","repo_kind":"listed","path":"lib/inference/generation.py","file_url":"https://github.com/lsc-1/pecola/blob/HEAD/lib/inference/generation.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"24c1d85c8d245fbb"}},{"code_sha256_prefix":"c5c3b0d2f5c9931c","entry":"get_model","repo":"lsc-1/pecola","repo_kind":"listed","path":"lib/networks.py","file_url":"https://github.com/lsc-1/pecola/blob/HEAD/lib/networks.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c5c3b0d2f5c9931c"}},{"code_sha256_prefix":"f70aa83668a8706c","entry":"get_ppl_confidence","repo":"lsc-1/pecola","repo_kind":"listed","path":"lib/inference/generation.py","file_url":"https://github.com/lsc-1/pecola/blob/HEAD/lib/inference/generation.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f70aa83668a8706c"}},{"code_sha256_prefix":"73808e9a70c072ec","entry":"get_tokenizer","repo":"lsc-1/pecola","repo_kind":"listed","path":"lib/networks.py","file_url":"https://github.com/lsc-1/pecola/blob/HEAD/lib/networks.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"73808e9a70c072ec"}},{"code_sha256_prefix":"518d9f2cda25ff12","entry":"roc","repo":"lsc-1/pecola","repo_kind":"listed","path":"lib/metrics.py","file_url":"https://github.com/lsc-1/pecola/blob/HEAD/lib/metrics.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"518d9f2cda25ff12"}},{"code_sha256_prefix":"85474e5c0d2f6525","entry":"take_mean_and_reshape","repo":"lsc-1/pecola","repo_kind":"listed","path":"lib/exp.py","file_url":"https://github.com/lsc-1/pecola/blob/HEAD/lib/exp.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"85474e5c0d2f6525"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}