{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/m4gt-bench-evaluation-benchmark-for-black-box","title":"M4GT-Bench: Evaluation Benchmark for Black-Box Machine-Generated Text Detection","arxiv_id":"2402.11175","date":"2024-02-17","proceeding":null,"authors":["Yuxia Wang","Jonibek Mansurov","Petar Ivanov","Jinyan Su","Artem Shelmanov","Akim Tsvigun","Osama Mohanned Afzal","Tarek Mahmoud","Giovanni Puccetti","Thomas Arnold","Alham Fikri Aji","Nizar Habash","Iryna Gurevych","Preslav Nakov"],"abstract":"The advent of Large Language Models (LLMs) has brought an unprecedented surge in machine-generated text (MGT) across diverse channels. This raises legitimate concerns about its potential misuse and societal implications. The need to identify and differentiate such content from genuine human-generated text is critical in combating disinformation, preserving the integrity of education and scientific fields, and maintaining trust in communication. In this work, we address this problem by introducing a new benchmark based on a multilingual, multi-domain, and multi-generator corpus of MGTs -- M4GT-Bench. The benchmark is compiled of three tasks: (1) mono-lingual and multi-lingual binary MGT detection; (2) multi-way detection where one need to identify, which particular model generated the text; and (3) mixed human-machine text detection, where a word boundary delimiting MGT from human-written content should be determined. On the developed benchmark, we have tested several MGT detection baselines and also conducted an evaluation of human performance. We see that obtaining good performance in MGT detection usually requires an access to the training data from the same domain and generators. The benchmark is available at https://github.com/mbzuai-nlp/M4GT-Bench.","url_abs":"https://arxiv.org/abs/2402.11175v2","url_pdf":"https://arxiv.org/pdf/2402.11175v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"m4gt-bench-evaluation-benchmark-for-black-box","repo_url":"https://github.com/mbzuai-nlp/m4gt-bench","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok"}}],"tasks":[{"task_slug":"task-2","task_name":"Task 2"},{"task_slug":"text-detection","task_name":"Text Detection"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2402.11175","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2402.11175"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/mbzuai-nlp/m4gt-bench","reach":{"status":"ok"}}],"summary":{"ran_draft_wrong":1,"ran":7,"unverified":2},"by_repo_kind":{"official":{"samples":10,"ran":8,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":10,"samples":[{"code_sha256_prefix":"c3dfa77d1a3ca4e1","entry":"analyze","repo":"mbzuai-nlp/m4gt-bench","repo_kind":"official","path":"src/GLTR/server.py","file_url":"https://github.com/mbzuai-nlp/m4gt-bench/blob/HEAD/src/GLTR/server.py","link_basis":"plan_row","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"c3dfa77d1a3ca4e1"}},{"code_sha256_prefix":"c05abc7d4fef8a91","entry":"count_top_k","repo":"mbzuai-nlp/m4gt-bench","repo_kind":"official","path":"src/extract_gltr_features.py","file_url":"https://github.com/mbzuai-nlp/m4gt-bench/blob/HEAD/src/extract_gltr_features.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"c05abc7d4fef8a91"}},{"code_sha256_prefix":"3c1eb3db9a05dc2a","entry":"get_data","repo":"mbzuai-nlp/m4gt-bench","repo_kind":"official","path":"src/transformer_detector.py","file_url":"https://github.com/mbzuai-nlp/m4gt-bench/blob/HEAD/src/transformer_detector.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"3c1eb3db9a05dc2a"}},{"code_sha256_prefix":"c15227bbe3011bf3","entry":"get_gltr_features","repo":"mbzuai-nlp/m4gt-bench","repo_kind":"official","path":"src/gltr_lr.py","file_url":"https://github.com/mbzuai-nlp/m4gt-bench/blob/HEAD/src/gltr_lr.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"c15227bbe3011bf3"}},{"code_sha256_prefix":"d12230a391190b64","entry":"gltr_payload_to_features","repo":"mbzuai-nlp/m4gt-bench","repo_kind":"official","path":"src/extract_gltr_features.py","file_url":"https://github.com/mbzuai-nlp/m4gt-bench/blob/HEAD/src/extract_gltr_features.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"d12230a391190b64"}},{"code_sha256_prefix":"59853f665d709fd2","entry":"preprocess_function","repo":"mbzuai-nlp/m4gt-bench","repo_kind":"official","path":"src/transformer_detector.py","file_url":"https://github.com/mbzuai-nlp/m4gt-bench/blob/HEAD/src/transformer_detector.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"59853f665d709fd2"}},{"code_sha256_prefix":"23dd1413a544be59","entry":"register_api","repo":"mbzuai-nlp/m4gt-bench","repo_kind":"official","path":"src/GLTR/backend/class_register.py","file_url":"https://github.com/mbzuai-nlp/m4gt-bench/blob/HEAD/src/GLTR/backend/class_register.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"23dd1413a544be59"}},{"code_sha256_prefix":"53711ba356067419","entry":"train","repo":"mbzuai-nlp/m4gt-bench","repo_kind":"official","path":"src/gltr_lr.py","file_url":"https://github.com/mbzuai-nlp/m4gt-bench/blob/HEAD/src/gltr_lr.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"53711ba356067419"}},{"code_sha256_prefix":"7be611d6e7fba121","entry":"top_k_logits","repo":"mbzuai-nlp/m4gt-bench","repo_kind":"official","path":"src/GLTR/backend/api.py","file_url":"https://github.com/mbzuai-nlp/m4gt-bench/blob/HEAD/src/GLTR/backend/api.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"7be611d6e7fba121"}},{"code_sha256_prefix":"cb4ff66520d69188","entry":"train","repo":"mbzuai-nlp/m4gt-bench","repo_kind":"official","path":"src/gltr_svm.py","file_url":"https://github.com/mbzuai-nlp/m4gt-bench/blob/HEAD/src/gltr_svm.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"cb4ff66520d69188"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}