{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/benchmarking-the-robustness-of-image","title":"WAVES: Benchmarking the Robustness of Image Watermarks","arxiv_id":"2401.08573","date":"2024-01-16","proceeding":null,"authors":["Bang An","Mucong Ding","Tahseen Rabbani","Aakriti Agrawal","Yuancheng Xu","ChengHao Deng","Sicheng Zhu","Abdirisak Mohamed","Yuxin Wen","Tom Goldstein","Furong Huang"],"abstract":"In the burgeoning age of generative AI, watermarks act as identifiers of provenance and artificial content. We present WAVES (Watermark Analysis Via Enhanced Stress-testing), a benchmark for assessing image watermark robustness, overcoming the limitations of current evaluation methods. WAVES integrates detection and identification tasks and establishes a standardized evaluation protocol comprised of a diverse range of stress tests. The attacks in WAVES range from traditional image distortions to advanced, novel variations of diffusive, and adversarial attacks. Our evaluation examines two pivotal dimensions: the degree of image quality degradation and the efficacy of watermark detection after attacks. Our novel, comprehensive evaluation reveals previously undetected vulnerabilities of several modern watermarking algorithms. We envision WAVES as a toolkit for the future development of robust watermarks. The project is available at https://wavesbench.github.io/","url_abs":"https://arxiv.org/abs/2401.08573v3","url_pdf":"https://arxiv.org/pdf/2401.08573v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"benchmarking-the-robustness-of-image","repo_url":"https://github.com/umd-huang-lab/waves","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":"benchmarking","task_name":"Benchmarking"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2401.08573","atlas_url":"https://app.syntology.ai/?focus=2401.08573","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2401.08573"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/umd-huang-lab/waves","reach":{"status":"ok"}}],"summary":{"ran_draft_wrong":3,"ran":7,"ran_honours":1,"ran_fixture":1,"unverified":9},"by_repo_kind":{"official":{"samples":19,"ran":10,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":21,"samples":[{"code_sha256_prefix":"cfd76fd0d89574a4","entry":"approx_standard_normal_cdf","repo":"umd-huang-lab/waves","repo_kind":"official","path":"guided_diffusion/losses.py","file_url":"https://github.com/umd-huang-lab/waves/blob/HEAD/guided_diffusion/losses.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"cfd76fd0d89574a4"}},{"code_sha256_prefix":"0e979ef06ee3d7df","entry":"check_file_existence","repo":"umd-huang-lab/waves","repo_kind":"official","path":"dev/find.py","file_url":"https://github.com/umd-huang-lab/waves/blob/HEAD/dev/find.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"0e979ef06ee3d7df"}},{"code_sha256_prefix":"cd33283d615fb3d7","entry":"discretized_gaussian_log_likelihood","repo":"umd-huang-lab/waves","repo_kind":"official","path":"guided_diffusion/losses.py","file_url":"https://github.com/umd-huang-lab/waves/blob/HEAD/guided_diffusion/losses.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"cd33283d615fb3d7"}},{"code_sha256_prefix":"1a946fd51f8d53d0","entry":"encode_array_to_string","repo":"umd-huang-lab/waves","repo_kind":"official","path":"dev/io.py","file_url":"https://github.com/umd-huang-lab/waves/blob/HEAD/dev/io.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"1a946fd51f8d53d0"}},{"code_sha256_prefix":"e849fa32ea51cc91","entry":"existence_operation","repo":"umd-huang-lab/waves","repo_kind":"official","path":"dev/find.py","file_url":"https://github.com/umd-huang-lab/waves/blob/HEAD/dev/find.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"e849fa32ea51cc91"}},{"code_sha256_prefix":"35752197857e37ec","entry":"existence_to_indices","repo":"umd-huang-lab/waves","repo_kind":"official","path":"dev/find.py","file_url":"https://github.com/umd-huang-lab/waves/blob/HEAD/dev/find.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"35752197857e37ec"}},{"code_sha256_prefix":"c27f397fb1d357ee","entry":"get_default_guided_diffusion_paras","repo":"umd-huang-lab/waves","repo_kind":"official","path":"guided_diffusion/model_utils.py","file_url":"https://github.com/umd-huang-lab/waves/blob/HEAD/guided_diffusion/model_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"c27f397fb1d357ee"}},{"code_sha256_prefix":"dae4eb02495b6bb2","entry":"load_json","repo":"umd-huang-lab/waves","repo_kind":"official","path":"dev/io.py","file_url":"https://github.com/umd-huang-lab/waves/blob/HEAD/dev/io.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"dae4eb02495b6bb2"}},{"code_sha256_prefix":"cf2798b666b231ca","entry":"normal_kl","repo":"umd-huang-lab/waves","repo_kind":"official","path":"guided_diffusion/losses.py","file_url":"https://github.com/umd-huang-lab/waves/blob/HEAD/guided_diffusion/losses.py","link_basis":"harvester_set","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"cf2798b666b231ca"}},{"code_sha256_prefix":"29e5e72bcd006dcd","entry":"spatial_average","repo":null,"repo_kind":null,"path":null,"file_url":null,"link_basis":"identical_code_first_harvested_elsewhere","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":null,"inline_ok":false,"mcp_get_code":{"code_sha256":"29e5e72bcd006dcd"}},{"code_sha256_prefix":"9f7402d7db928d89","entry":"style_progress_dataframe","repo":"umd-huang-lab/waves","repo_kind":"official","path":"dev/plot.py","file_url":"https://github.com/umd-huang-lab/waves/blob/HEAD/dev/plot.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"9f7402d7db928d89"}},{"code_sha256_prefix":"8cafd18b57a4d171","entry":"upsample","repo":null,"repo_kind":null,"path":null,"file_url":null,"link_basis":"identical_code_first_harvested_elsewhere","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":null,"inline_ok":false,"mcp_get_code":{"code_sha256":"8cafd18b57a4d171"}},{"code_sha256_prefix":"567eac518af56dd7","entry":"aggregate_comparison_dataframe","repo":"umd-huang-lab/waves","repo_kind":"official","path":"dev/plot.py","file_url":"https://github.com/umd-huang-lab/waves/blob/HEAD/dev/plot.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"567eac518af56dd7"}},{"code_sha256_prefix":"576e24bf26be2f7d","entry":"compare_dicts","repo":"umd-huang-lab/waves","repo_kind":"official","path":"dev/io.py","file_url":"https://github.com/umd-huang-lab/waves/blob/HEAD/dev/io.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"576e24bf26be2f7d"}},{"code_sha256_prefix":"74f41da5353d6e73","entry":"get_distances_from_json","repo":"umd-huang-lab/waves","repo_kind":"official","path":"dev/parse.py","file_url":"https://github.com/umd-huang-lab/waves/blob/HEAD/dev/parse.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"74f41da5353d6e73"}},{"code_sha256_prefix":"cbfe008d27374dc6","entry":"get_example_from_json","repo":"umd-huang-lab/waves","repo_kind":"official","path":"dev/parse.py","file_url":"https://github.com/umd-huang-lab/waves/blob/HEAD/dev/parse.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"cbfe008d27374dc6"}},{"code_sha256_prefix":"204386be886582f9","entry":"get_performance","repo":"umd-huang-lab/waves","repo_kind":"official","path":"dev/aggregate.py","file_url":"https://github.com/umd-huang-lab/waves/blob/HEAD/dev/aggregate.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"204386be886582f9"}},{"code_sha256_prefix":"0b911800b7aa751f","entry":"get_performance_from_jsons","repo":"umd-huang-lab/waves","repo_kind":"official","path":"dev/aggregate.py","file_url":"https://github.com/umd-huang-lab/waves/blob/HEAD/dev/aggregate.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"0b911800b7aa751f"}},{"code_sha256_prefix":"9f218ff600660474","entry":"get_progress_from_json","repo":"umd-huang-lab/waves","repo_kind":"official","path":"dev/parse.py","file_url":"https://github.com/umd-huang-lab/waves/blob/HEAD/dev/parse.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"9f218ff600660474"}},{"code_sha256_prefix":"ede364a9709ec4a4","entry":"get_single_quality_from_jsons","repo":"umd-huang-lab/waves","repo_kind":"official","path":"dev/aggregate.py","file_url":"https://github.com/umd-huang-lab/waves/blob/HEAD/dev/aggregate.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"ede364a9709ec4a4"}},{"code_sha256_prefix":"85305761b9eb49b3","entry":"load_guided_diffusion_model","repo":"umd-huang-lab/waves","repo_kind":"official","path":"guided_diffusion/model_utils.py","file_url":"https://github.com/umd-huang-lab/waves/blob/HEAD/guided_diffusion/model_utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"85305761b9eb49b3"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}