{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/gigacheck-detecting-llm-generated-content","title":"GigaCheck: Detecting LLM-generated Content","arxiv_id":"2410.23728","date":"2024-10-31","proceeding":null,"authors":["Irina Tolstykh","Aleksandra Tsybina","Sergey Yakubson","Aleksandr Gordeev","Vladimir Dokholyan","Maksim Kuprashevich"],"abstract":"With the increasing quality and spread of LLM-based assistants, the amount of LLM-generated content is growing rapidly. In many cases and tasks, such texts are already indistinguishable from those written by humans, and the quality of generation tends to only increase. At the same time, detection methods are developing more slowly, making it challenging to prevent misuse of generative AI technologies. In this work, we investigate the task of generated text detection by proposing the GigaCheck. Our research explores two approaches: (i) distinguishing human-written texts from LLM-generated ones, and (ii) detecting LLM-generated intervals in Human-Machine collaborative texts. For the first task, our approach utilizes a general-purpose LLM, leveraging its extensive language abilities to fine-tune efficiently for the downstream task of LLM-generated text detection, achieving high performance even with limited data. For the second task, we propose a novel approach that combines computer vision and natural language processing techniques. Specifically, we use a fine-tuned general-purpose LLM in conjunction with a DETR-like detection model, adapted from computer vision, to localize AI-generated intervals within text. We evaluate the GigaCheck on five classification datasets with English texts and three datasets designed for Human-Machine collaborative text analysis. Our results demonstrate that GigaCheck outperforms previous methods, even in out-of-distribution settings, establishing a strong baseline across all datasets.","url_abs":"https://arxiv.org/abs/2410.23728v2","url_pdf":"https://arxiv.org/pdf/2410.23728v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[],"tasks":[{"task_slug":"binary-text-classification","task_name":"Binary text classification"},{"task_slug":"boundary-detection","task_name":"Boundary Detection"},{"task_slug":"llm-generated-text-detection","task_name":"LLM-generated Text Detection"},{"task_slug":"text-classification","task_name":"Text Classification"},{"task_slug":"text-detection","task_name":"Text Detection"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/binary-text-classification-on-ghostbuster-all","task":"Binary text classification","dataset":"Ghostbuster (All Domains)","model":"GigaCheck (Mistral-7B)","rank_in_archive_order":1,"of":2,"metrics":{"F1 score":"1.0"},"uses_additional_data":false},{"leaderboard":"/sota/binary-text-classification-on-mage-arbitrary","task":"Binary text classification","dataset":"MAGE (Arbitrary-domains & Arbitrary-models)","model":"GigaCheck (Mistral-7B)","rank_in_archive_order":1,"of":2,"metrics":{"Average Recall":"0.9611"},"uses_additional_data":false},{"leaderboard":"/sota/binary-text-classification-on-mixset-binary","task":"Binary text classification","dataset":"MixSet (Binary)","model":"GigaCheck (Mistral-7B)","rank_in_archive_order":1,"of":2,"metrics":{"F1 score":"0.99"},"uses_additional_data":false},{"leaderboard":"/sota/binary-text-classification-on-turingbench-1","task":"Binary text classification","dataset":"TURINGBENCH (Turing Test, FAIR_wmt20)","model":"GigaCheck (Mistral-7B)","rank_in_archive_order":1,"of":2,"metrics":{"F1 score":"0.9966"},"uses_additional_data":false},{"leaderboard":"/sota/binary-text-classification-on-turingbench","task":"Binary text classification","dataset":"TURINGBENCH (Turing Test, GPT-3)","model":"GigaCheck (Mistral-7B)","rank_in_archive_order":1,"of":2,"metrics":{"F1 score":"0.9709"},"uses_additional_data":false},{"leaderboard":"/sota/binary-text-classification-on-tweepfake","task":"Binary text classification","dataset":"TweepFake","model":"GigaCheck (Mistral-7B)","rank_in_archive_order":1,"of":2,"metrics":{"Accuracy (%)":"94.3","F1 score":"0.942"},"uses_additional_data":false},{"leaderboard":"/sota/boundary-detection-on-coauthor","task":"Boundary Detection","dataset":"CoAuthor","model":"GigaCheck (Mistral-7B-v0.3)","rank_in_archive_order":1,"of":3,"metrics":{"Cohen’s Kappa score":"0.4158"},"uses_additional_data":false},{"leaderboard":"/sota/boundary-detection-on-coauthor","task":"Boundary Detection","dataset":"CoAuthor","model":"GigaCheck (DN-DAB-DETR)","rank_in_archive_order":3,"of":3,"metrics":{"Cohen’s Kappa score":"0.1885"},"uses_additional_data":false},{"leaderboard":"/sota/boundary-detection-on-roft","task":"Boundary Detection","dataset":"RoFT","model":"GigaCheck (DN-DAB-DETR)","rank_in_archive_order":1,"of":4,"metrics":{"Accuracy (%)":"64.63","MSE":"1.51"},"uses_additional_data":false},{"leaderboard":"/sota/boundary-detection-on-roft-chatgpt","task":"Boundary Detection","dataset":"RoFT-chatgpt","model":"GigaCheck (DN-DAB-DETR)","rank_in_archive_order":1,"of":4,"metrics":{"Accuracy (%)":"67.65","MSE":"1.03"},"uses_additional_data":false},{"leaderboard":"/sota/boundary-detection-on-tribert-in-domain","task":"Boundary Detection","dataset":"TriBERT (in-domain)","model":"GigaCheck (DN-DAB-DETR)","rank_in_archive_order":1,"of":2,"metrics":{"F1@3":"0.646"},"uses_additional_data":false}],"syntology":{"atlas_url":null,"mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}