{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/real-or-fake-text-investigating-human-ability","title":"Real or Fake Text?: Investigating Human Ability to Detect Boundaries Between Human-Written and Machine-Generated Text","arxiv_id":"2212.12672","date":"2022-12-24","proceeding":null,"authors":["Liam Dugan","Daphne Ippolito","Arun Kirubarajan","Sherry Shi","Chris Callison-Burch"],"abstract":"As text generated by large language models proliferates, it becomes vital to understand how humans engage with such text, and whether or not they are able to detect when the text they are reading did not originate with a human writer. Prior work on human detection of generated text focuses on the case where an entire passage is either human-written or machine-generated. In this paper, we study a more realistic setting where text begins as human-written and transitions to being generated by state-of-the-art neural language models. We show that, while annotators often struggle at this task, there is substantial variance in annotator skill and that given proper incentives, annotators can improve at this task over time. Furthermore, we conduct a detailed comparison study and analyze how a variety of variables (model size, decoding strategy, fine-tuning, prompt genre, etc.) affect human detection performance. Finally, we collect error annotations from our participants and use them to show that certain textual genres influence models to make different types of errors and that certain sentence-level features correlate highly with annotator selection. We release the RoFT dataset: a collection of over 21,000 human annotations paired with error classifications to encourage future work in human detection and evaluation of generated text.","url_abs":"https://arxiv.org/abs/2212.12672v1","url_pdf":"https://arxiv.org/pdf/2212.12672v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"real-or-fake-text-investigating-human-ability","repo_url":"https://github.com/liamdugan/human-detection","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"real-or-fake-text-investigating-human-ability","repo_url":"https://github.com/silversolver/ai_boundary_detection","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":"human-detection","task_name":"Human Detection"},{"task_slug":"sentence","task_name":"Sentence"}],"methods":[],"datasets_introduced":[{"slug":"roft","name":"RoFT","full_name":"Real or Fake Text"}],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2212.12672","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2212.12672"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/silversolver/ai_boundary_detection","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/liamdugan/human-detection","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"unverified":14},"by_repo_kind":{"official":{"samples":14,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"0728fceba9b35f2c","entry":"clean","repo":"liamdugan/human-detection","repo_kind":"official","path":"generation/process_datasets/process-NYT.py","file_url":"https://github.com/liamdugan/human-detection/blob/HEAD/generation/process_datasets/process-NYT.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"0728fceba9b35f2c"}},{"code_sha256_prefix":"036b76d1aab8ff74","entry":"content_filter","repo":"liamdugan/human-detection","repo_kind":"official","path":"generation/content_filter.py","file_url":"https://github.com/liamdugan/human-detection/blob/HEAD/generation/content_filter.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"036b76d1aab8ff74"}},{"code_sha256_prefix":"e3b1a763e169ffeb","entry":"cutoff_prompt","repo":"liamdugan/human-detection","repo_kind":"official","path":"generation/inference_scripts/roft_ctrl_generator.py","file_url":"https://github.com/liamdugan/human-detection/blob/HEAD/generation/inference_scripts/roft_ctrl_generator.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e3b1a763e169ffeb"}},{"code_sha256_prefix":"3eb12ffcbdba4d96","entry":"determine_outfile","repo":"liamdugan/human-detection","repo_kind":"official","path":"generation/process_datasets/process-Reddit.py","file_url":"https://github.com/liamdugan/human-detection/blob/HEAD/generation/process_datasets/process-Reddit.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"3eb12ffcbdba4d96"}},{"code_sha256_prefix":"188bfe6cd3c544b2","entry":"download_sampling_file","repo":"liamdugan/human-detection","repo_kind":"official","path":"generation/finetuning_scripts/roft_gpt2_finetuning.py","file_url":"https://github.com/liamdugan/human-detection/blob/HEAD/generation/finetuning_scripts/roft_gpt2_finetuning.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"188bfe6cd3c544b2"}},{"code_sha256_prefix":"3d784c8b761ec9af","entry":"fix_quotation_marks","repo":"liamdugan/human-detection","repo_kind":"official","path":"generation/inference_scripts/roft_gpt2_generator.py","file_url":"https://github.com/liamdugan/human-detection/blob/HEAD/generation/inference_scripts/roft_gpt2_generator.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"3d784c8b761ec9af"}},{"code_sha256_prefix":"e7e6275255e634a0","entry":"get_outfile","repo":"liamdugan/human-detection","repo_kind":"official","path":"generation/process_datasets/process-NYT.py","file_url":"https://github.com/liamdugan/human-detection/blob/HEAD/generation/process_datasets/process-NYT.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e7e6275255e634a0"}},{"code_sha256_prefix":"ab69574664018456","entry":"is_percentage","repo":"liamdugan/human-detection","repo_kind":"official","path":"processing/filter_data.py","file_url":"https://github.com/liamdugan/human-detection/blob/HEAD/processing/filter_data.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ab69574664018456"}},{"code_sha256_prefix":"4e8d6dbed9143eea","entry":"makedirs","repo":"liamdugan/human-detection","repo_kind":"official","path":"generation/process_datasets/process-NYT.py","file_url":"https://github.com/liamdugan/human-detection/blob/HEAD/generation/process_datasets/process-NYT.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"4e8d6dbed9143eea"}},{"code_sha256_prefix":"8a6414ef46ee5bc8","entry":"parse_file","repo":"liamdugan/human-detection","repo_kind":"official","path":"generation/process_datasets/process-Speeches.py","file_url":"https://github.com/liamdugan/human-detection/blob/HEAD/generation/process_datasets/process-Speeches.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"8a6414ef46ee5bc8"}},{"code_sha256_prefix":"dc5677aaab1a689d","entry":"remap_familiarity_labels","repo":"liamdugan/human-detection","repo_kind":"official","path":"processing/process_database_dump.py","file_url":"https://github.com/liamdugan/human-detection/blob/HEAD/processing/process_database_dump.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"dc5677aaab1a689d"}},{"code_sha256_prefix":"03dae20394246fc3","entry":"remap_genre_fam_labels","repo":"liamdugan/human-detection","repo_kind":"official","path":"processing/process_database_dump.py","file_url":"https://github.com/liamdugan/human-detection/blob/HEAD/processing/process_database_dump.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"03dae20394246fc3"}},{"code_sha256_prefix":"e4ec01aaf7c17a31","entry":"roft_prompts_json_to_dataset","repo":"liamdugan/human-detection","repo_kind":"official","path":"generation/finetuning_scripts/roft_gpt2_finetuning.py","file_url":"https://github.com/liamdugan/human-detection/blob/HEAD/generation/finetuning_scripts/roft_gpt2_finetuning.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e4ec01aaf7c17a31"}},{"code_sha256_prefix":"6e414eceb5070e90","entry":"sliding_window","repo":"liamdugan/human-detection","repo_kind":"official","path":"processing/filter_data.py","file_url":"https://github.com/liamdugan/human-detection/blob/HEAD/processing/filter_data.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"6e414eceb5070e90"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}