{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/people-make-better-edits-measuring-the","title":"People Make Better Edits: Measuring the Efficacy of LLM-Generated Counterfactually Augmented Data for Harmful Language Detection","arxiv_id":"2311.01270","date":"2023-11-02","proceeding":null,"authors":["Indira Sen","Dennis Assenmacher","Mattia Samory","Isabelle Augenstein","Wil van der Aalst","Claudia Wagner"],"abstract":"NLP models are used in a variety of critical social computing tasks, such as detecting sexist, racist, or otherwise hateful content. Therefore, it is imperative that these models are robust to spurious features. Past work has attempted to tackle such spurious features using training data augmentation, including Counterfactually Augmented Data (CADs). CADs introduce minimal changes to existing training data points and flip their labels; training on them may reduce model dependency on spurious features. However, manually generating CADs can be time-consuming and expensive. Hence in this work, we assess if this task can be automated using generative NLP models. We automatically generate CADs using Polyjuice, ChatGPT, and Flan-T5, and evaluate their usefulness in improving model robustness compared to manually-generated CADs. By testing both model performance on multiple out-of-domain test sets and individual data point efficacy, our results show that while manual CADs are still the most effective, CADs generated by ChatGPT come a close second. One key reason for the lower performance of automated methods is that the changes they introduce are often insufficient to flip the original label.","url_abs":"https://arxiv.org/abs/2311.01270v3","url_pdf":"https://arxiv.org/pdf/2311.01270v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"people-make-better-edits-measuring-the","repo_url":"https://github.com/indiiigo/automatedcad","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok"}}],"tasks":[{"task_slug":"data-augmentation","task_name":"Data Augmentation"}],"methods":[{"method_slug":"flip","method_name":"FLIP"},{"method_slug":"flan-t5","method_name":"Flan-T5"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2311.01270","atlas_url":"https://app.syntology.ai/?focus=2311.01270","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2311.01270"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/indiiigo/automatedcad","reach":{"status":"ok"}}],"summary":{"ran":1,"unverified":1},"by_repo_kind":{"official":{"samples":2,"ran":1,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":2,"samples":[{"code_sha256_prefix":"c0d5fd39c5abb5a8","entry":"find_similarity","repo":"indiiigo/automatedcad","repo_kind":"official","path":"code/rq2_which_cad/find_sim.py","file_url":"https://github.com/indiiigo/automatedcad/blob/HEAD/code/rq2_which_cad/find_sim.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"c0d5fd39c5abb5a8"}},{"code_sha256_prefix":"cca9d957c36e5fdd","entry":"sent_tokenize","repo":"indiiigo/automatedcad","repo_kind":"official","path":"code/rq2_which_cad/embed_docs.py","file_url":"https://github.com/indiiigo/automatedcad/blob/HEAD/code/rq2_which_cad/embed_docs.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"cca9d957c36e5fdd"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}