{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/performance-trade-offs-of-watermarking-large","title":"Downstream Trade-offs of a Family of Text Watermarks","arxiv_id":"2311.09816","date":"2023-11-16","proceeding":null,"authors":["Anirudh Ajith","Sameer Singh","Danish Pruthi"],"abstract":"Watermarking involves implanting an imperceptible signal into generated text that can later be detected via statistical tests. A prominent family of watermarking strategies for LLMs embeds this signal by upsampling a (pseudorandomly-chosen) subset of tokens at every generation step. However, such signals alter the model's output distribution and can have unintended effects on its downstream performance. In this work, we evaluate the performance of LLMs watermarked using three different strategies over a diverse suite of tasks including those cast as k-class classification (CLS), multiple choice question answering (MCQ), short-form generation (e.g., open-ended question answering) and long-form generation (e.g., translation) tasks. We find that watermarks (under realistic hyperparameters) can cause significant drops in LLMs' effective utility across all tasks. We observe drops of 10 to 20% in CLS tasks in the average case, which shoot up to 100% in the worst case. We notice degradations of about 7% in MCQ tasks, 10-15% in short-form generation, and 5-15% in long-form generation tasks. Our findings highlight the trade-offs that users should be cognizant of when using watermarked models.","url_abs":"https://arxiv.org/abs/2311.09816v2","url_pdf":"https://arxiv.org/pdf/2311.09816v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"performance-trade-offs-of-watermarking-large","repo_url":"https://github.com/flair-iisc/watermark_tradeoffs","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":"form","task_name":"Form"},{"task_slug":"language-modelling","task_name":"Language Modelling"},{"task_slug":"misinformation","task_name":"Misinformation"},{"task_slug":"multiple-choice","task_name":"Multiple-choice"},{"task_slug":"natural-language-inference","task_name":"Natural Language Inference"},{"task_slug":"open-question","task_name":"Open-Ended Question Answering"},{"task_slug":"question-answering","task_name":"Question Answering"},{"task_slug":"text-classification","task_name":"Text Classification"},{"task_slug":"translation","task_name":"Translation"},{"task_slug":"text-classification-1","task_name":"text-classification"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2311.09816","atlas_url":"https://app.syntology.ai/?focus=2311.09816","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2311.09816"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/flair-iisc/watermark_tradeoffs","reach":{"status":"ok"}}],"summary":{"ran":5,"unverified":1},"by_repo_kind":{"official":{"samples":6,"ran":5,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":6,"samples":[{"code_sha256_prefix":"08cb85fd2c4518cc","entry":"bleu_score","repo":"flair-iisc/watermark_tradeoffs","repo_kind":"official","path":"evaluation/utils.py","file_url":"https://github.com/flair-iisc/watermark_tradeoffs/blob/HEAD/evaluation/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"08cb85fd2c4518cc"}},{"code_sha256_prefix":"3bbc821ec9ed3957","entry":"multiway_start_idx","repo":"flair-iisc/watermark_tradeoffs","repo_kind":"official","path":"evaluation/utils.py","file_url":"https://github.com/flair-iisc/watermark_tradeoffs/blob/HEAD/evaluation/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"3bbc821ec9ed3957"}},{"code_sha256_prefix":"e5c4406aa5ea41c5","entry":"read_pareto_optimal_gamma_delta","repo":"flair-iisc/watermark_tradeoffs","repo_kind":"official","path":"evaluation/downstream_eval.py","file_url":"https://github.com/flair-iisc/watermark_tradeoffs/blob/HEAD/evaluation/downstream_eval.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"e5c4406aa5ea41c5"}},{"code_sha256_prefix":"061b8c77b92b1911","entry":"two_way_start_idx","repo":"flair-iisc/watermark_tradeoffs","repo_kind":"official","path":"evaluation/utils.py","file_url":"https://github.com/flair-iisc/watermark_tradeoffs/blob/HEAD/evaluation/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"061b8c77b92b1911"}},{"code_sha256_prefix":"2c19933145c388af","entry":"watermark_name_from_alg_name","repo":"flair-iisc/watermark_tradeoffs","repo_kind":"official","path":"watermark/auto_watermark.py","file_url":"https://github.com/flair-iisc/watermark_tradeoffs/blob/HEAD/watermark/auto_watermark.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"2c19933145c388af"}},{"code_sha256_prefix":"7840b92e19eb6f7d","entry":"load_dataset","repo":"flair-iisc/watermark_tradeoffs","repo_kind":"official","path":"evaluation/dataset.py","file_url":"https://github.com/flair-iisc/watermark_tradeoffs/blob/HEAD/evaluation/dataset.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"7840b92e19eb6f7d"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}