{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/few-shot-detection-of-machine-generated-text","title":"Few-Shot Detection of Machine-Generated Text using Style Representations","arxiv_id":"2401.06712","date":"2024-01-12","proceeding":null,"authors":["Rafael Rivera Soto","Kailin Koch","Aleem Khan","Barry Chen","Marcus Bishop","Nicholas Andrews"],"abstract":"The advent of instruction-tuned language models that convincingly mimic human writing poses a significant risk of abuse. However, such abuse may be counteracted with the ability to detect whether a piece of text was composed by a language model rather than a human author. Some previous approaches to this problem have relied on supervised methods by training on corpora of confirmed human- and machine- written documents. Unfortunately, model under-specification poses an unavoidable challenge for neural network-based detectors, making them brittle in the face of data shifts, such as the release of newer language models producing still more fluent text than the models used to train the detectors. Other approaches require access to the models that may have generated a document in question, which is often impractical. In light of these challenges, we pursue a fundamentally different approach not relying on samples from language models of concern at training time. Instead, we propose to leverage representations of writing style estimated from human-authored text. Indeed, we find that features effective at distinguishing among human authors are also effective at distinguishing human from machine authors, including state-of-the-art large language models like Llama-2, ChatGPT, and GPT-4. Furthermore, given a handful of examples composed by each of several specific language models of interest, our approach affords the ability to predict which model generated a given document. The code and data to reproduce our experiments are available at https://github.com/LLNL/LUAR/tree/main/fewshot_iclr2024.","url_abs":"https://arxiv.org/abs/2401.06712v3","url_pdf":"https://arxiv.org/pdf/2401.06712v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"few-shot-detection-of-machine-generated-text","repo_url":"https://github.com/llnl/luar","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"language-modelling","task_name":"Language Modelling"}],"methods":[{"method_slug":"absolute-position-encodings","method_name":"Absolute Position Encodings"},{"method_slug":"adam","method_name":"Adam"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"bpe","method_name":"BPE"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"gpt-4","method_name":"GPT-4"},{"method_slug":"label-smoothing","method_name":"Label Smoothing"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"position-wise-feed-forward-layer","method_name":"Position-Wise Feed-Forward Layer"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"transformer","method_name":"Transformer"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2401.06712","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2401.06712"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/llnl/luar","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran":5,"unverified":1},"by_repo_kind":{"official":{"samples":6,"ran":5,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"7811eb1fab7ee8c5","entry":"create_batches","repo":"llnl/luar","repo_kind":"official","path":"machine_detection/machine_utils.py","file_url":"https://github.com/llnl/luar/blob/HEAD/machine_detection/machine_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"7811eb1fab7ee8c5"}},{"code_sha256_prefix":"c330cda867fef7af","entry":"extract_luar_embeddings","repo":"llnl/luar","repo_kind":"official","path":"machine_detection/machine_utils.py","file_url":"https://github.com/llnl/luar/blob/HEAD/machine_detection/machine_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"c330cda867fef7af"}},{"code_sha256_prefix":"38a42652b6f8061b","entry":"get_data","repo":"llnl/luar","repo_kind":"official","path":"machine_detection/machine_utils.py","file_url":"https://github.com/llnl/luar/blob/HEAD/machine_detection/machine_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"38a42652b6f8061b"}},{"code_sha256_prefix":"a65281ae25eff699","entry":"singleton","repo":"llnl/luar","repo_kind":"official","path":"fewshot_iclr2024/file_utils.py","file_url":"https://github.com/llnl/luar/blob/HEAD/fewshot_iclr2024/file_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"a65281ae25eff699"}},{"code_sha256_prefix":"a3e19197e69c1a89","entry":"tokenize_function","repo":"llnl/luar","repo_kind":"official","path":"fewshot_iclr2024/baseline_training/finetune_roberta.py","file_url":"https://github.com/llnl/luar/blob/HEAD/fewshot_iclr2024/baseline_training/finetune_roberta.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"a3e19197e69c1a89"}},{"code_sha256_prefix":"61923b3971011953","entry":"optuna_hp_space","repo":"llnl/luar","repo_kind":"official","path":"fewshot_iclr2024/baseline_training/finetune_roberta.py","file_url":"https://github.com/llnl/luar/blob/HEAD/fewshot_iclr2024/baseline_training/finetune_roberta.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"61923b3971011953"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}