{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/llm-as-a-coauthor-the-challenges-of-detecting","title":"LLM-as-a-Coauthor: Can Mixed Human-Written and Machine-Generated Text Be Detected?","arxiv_id":"2401.05952","date":"2024-01-11","proceeding":null,"authors":["Qihui Zhang","Chujie Gao","Dongping Chen","Yue Huang","Yixin Huang","Zhenyang Sun","Shilin Zhang","Weiye Li","Zhengyan Fu","Yao Wan","Lichao Sun"],"abstract":"With the rapid development and widespread application of Large Language Models (LLMs), the use of Machine-Generated Text (MGT) has become increasingly common, bringing with it potential risks, especially in terms of quality and integrity in fields like news, education, and science. Current research mainly focuses on purely MGT detection without adequately addressing mixed scenarios, including AI-revised Human-Written Text (HWT) or human-revised MGT. To tackle this challenge, we define mixtext, a form of mixed text involving both AI and human-generated content. Then, we introduce MixSet, the first dataset dedicated to studying these mixtext scenarios. Leveraging MixSet, we executed comprehensive experiments to assess the efficacy of prevalent MGT detectors in handling mixtext situations, evaluating their performance in terms of effectiveness, robustness, and generalization. Our findings reveal that existing detectors struggle to identify mixtext, particularly in dealing with subtle modifications and style adaptability. This research underscores the urgent need for more fine-grain detectors tailored for mixtext, offering valuable insights for future research. Code and Models are available at https://github.com/Dongping-Chen/MixSet.","url_abs":"https://arxiv.org/abs/2401.05952v2","url_pdf":"https://arxiv.org/pdf/2401.05952v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"llm-as-a-coauthor-the-challenges-of-detecting","repo_url":"https://github.com/dongping-chen/mixset","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}},{"paper_slug":"llm-as-a-coauthor-the-challenges-of-detecting","repo_url":"https://github.com/dongping-chen/mllm-as-a-judge","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":"binary-text-classification","task_name":"Binary text classification"}],"methods":[{"method_slug":"mixtext","method_name":"MixText"}],"datasets_introduced":[{"slug":"mixset","name":"MixSet","full_name":"Mixcase Dataset"}],"methods_introduced":[],"results":[{"leaderboard":"/sota/binary-text-classification-on-mixset-binary","task":"Binary text classification","dataset":"MixSet (Binary)","model":"Radar","rank_in_archive_order":2,"of":2,"metrics":{"F1 score":"0.876"},"uses_additional_data":false}],"syntology":{"syntology_url":"https://syntology.ai/paper/2401.05952","atlas_url":"https://app.syntology.ai/?focus=2401.05952","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2401.05952"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/dongping-chen/mixset","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/dongping-chen/mllm-as-a-judge","reach":{"status":"ok"}}],"summary":{"ran":6,"unverified":10},"by_repo_kind":{"official":{"samples":16,"ran":6,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":16,"samples":[{"code_sha256_prefix":"ea13cd1bc8c737e1","entry":"filter_test_data","repo":"dongping-chen/mixset","repo_kind":"official","path":"methods/utils.py","file_url":"https://github.com/dongping-chen/mixset/blob/HEAD/methods/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"ea13cd1bc8c737e1"}},{"code_sha256_prefix":"f83e125c58abd645","entry":"get_ll","repo":"dongping-chen/mixset","repo_kind":"official","path":"methods/metric_based.py","file_url":"https://github.com/dongping-chen/mixset/blob/HEAD/methods/metric_based.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"f83e125c58abd645"}},{"code_sha256_prefix":"68c8b968f55907b7","entry":"get_supervised_model_prediction_multi_classes","repo":"dongping-chen/mixset","repo_kind":"official","path":"methods/supervised.py","file_url":"https://github.com/dongping-chen/mixset/blob/HEAD/methods/supervised.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"68c8b968f55907b7"}},{"code_sha256_prefix":"416e7dabab64e070","entry":"select_train_data","repo":"dongping-chen/mixset","repo_kind":"official","path":"methods/utils.py","file_url":"https://github.com/dongping-chen/mixset/blob/HEAD/methods/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"416e7dabab64e070"}},{"code_sha256_prefix":"58ba64be72bc1b9e","entry":"timeit","repo":"dongping-chen/mixset","repo_kind":"official","path":"methods/utils.py","file_url":"https://github.com/dongping-chen/mixset/blob/HEAD/methods/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"58ba64be72bc1b9e"}},{"code_sha256_prefix":"639b00cb6787d6d0","entry":"tokenize_and_mask","repo":"dongping-chen/mixset","repo_kind":"official","path":"methods/detectgpt.py","file_url":"https://github.com/dongping-chen/mixset/blob/HEAD/methods/detectgpt.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"639b00cb6787d6d0"}},{"code_sha256_prefix":"23c3416429f69fee","entry":"count_masks","repo":"dongping-chen/mixset","repo_kind":"official","path":"methods/detectgpt.py","file_url":"https://github.com/dongping-chen/mixset/blob/HEAD/methods/detectgpt.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":"OPAQUE_PARAMS","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"23c3416429f69fee"}},{"code_sha256_prefix":"23449ffb5d7ab7b0","entry":"evaluate_model","repo":"dongping-chen/mixset","repo_kind":"official","path":"methods/radar.py","file_url":"https://github.com/dongping-chen/mixset/blob/HEAD/methods/radar.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"23449ffb5d7ab7b0"}},{"code_sha256_prefix":"8e3d09aab30cff33","entry":"evaluate_sentinel","repo":"dongping-chen/mixset","repo_kind":"official","path":"methods/sentinel.py","file_url":"https://github.com/dongping-chen/mixset/blob/HEAD/methods/sentinel.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"8e3d09aab30cff33"}},{"code_sha256_prefix":"c1d751eac084bac6","entry":"get_lls","repo":"dongping-chen/mixset","repo_kind":"official","path":"methods/metric_based.py","file_url":"https://github.com/dongping-chen/mixset/blob/HEAD/methods/metric_based.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"c1d751eac084bac6"}},{"code_sha256_prefix":"ec539267c749a0cc","entry":"get_rank","repo":"dongping-chen/mixset","repo_kind":"official","path":"methods/metric_based.py","file_url":"https://github.com/dongping-chen/mixset/blob/HEAD/methods/metric_based.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"ec539267c749a0cc"}},{"code_sha256_prefix":"ffed340b4b1b0302","entry":"get_supervised_model_prediction","repo":"dongping-chen/mixset","repo_kind":"official","path":"methods/supervised.py","file_url":"https://github.com/dongping-chen/mixset/blob/HEAD/methods/supervised.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"ffed340b4b1b0302"}},{"code_sha256_prefix":"d6120a808c697802","entry":"process_dataset","repo":"dongping-chen/mixset","repo_kind":"official","path":"dataset_loader.py","file_url":"https://github.com/dongping-chen/mixset/blob/HEAD/dataset_loader.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"d6120a808c697802"}},{"code_sha256_prefix":"c5d3444c3ea71db9","entry":"process_spaces","repo":"dongping-chen/mixset","repo_kind":"official","path":"dataset_loader.py","file_url":"https://github.com/dongping-chen/mixset/blob/HEAD/dataset_loader.py","link_basis":"plan_row","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"c5d3444c3ea71db9"}},{"code_sha256_prefix":"dd7573db30a9bd4e","entry":"process_text_truthfulqa_adv","repo":"dongping-chen/mixset","repo_kind":"official","path":"dataset_loader.py","file_url":"https://github.com/dongping-chen/mixset/blob/HEAD/dataset_loader.py","link_basis":"plan_row","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"dd7573db30a9bd4e"}},{"code_sha256_prefix":"3b992344e4b4f98b","entry":"replace_masks","repo":"dongping-chen/mixset","repo_kind":"official","path":"methods/detectgpt.py","file_url":"https://github.com/dongping-chen/mixset/blob/HEAD/methods/detectgpt.py","link_basis":"plan_row","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"3b992344e4b4f98b"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}