{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/on-discrete-prompt-optimization-for-diffusion","title":"On Discrete Prompt Optimization for Diffusion Models","arxiv_id":"2407.01606","date":"2024-06-27","proceeding":null,"authors":["Ruochen Wang","Ting Liu","Cho-Jui Hsieh","Boqing Gong"],"abstract":"This paper introduces the first gradient-based framework for prompt optimization in text-to-image diffusion models. We formulate prompt engineering as a discrete optimization problem over the language space. Two major challenges arise in efficiently finding a solution to this problem: (1) Enormous Domain Space: Setting the domain to the entire language space poses significant difficulty to the optimization process. (2) Text Gradient: Efficiently computing the text gradient is challenging, as it requires backpropagating through the inference steps of the diffusion model and a non-differentiable embedding lookup table. Beyond the problem formulation, our main technical contributions lie in solving the above challenges. First, we design a family of dynamically generated compact subspaces comprised of only the most relevant words to user input, substantially restricting the domain space. Second, we introduce \"Shortcut Text Gradient\" -- an effective replacement for the text gradient that can be obtained with constant memory and runtime. Empirical evaluation on prompts collected from diverse sources (DiffusionDB, ChatGPT, COCO) suggests that our method can discover prompts that substantially improve (prompt enhancement) or destroy (adversarial attack) the faithfulness of images generated by the text-to-image diffusion model.","url_abs":"https://arxiv.org/abs/2407.01606v1","url_pdf":"https://arxiv.org/pdf/2407.01606v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"on-discrete-prompt-optimization-for-diffusion","repo_url":"https://github.com/ruocwang/dpo-diffusion","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"adversarial-attack","task_name":"Adversarial Attack"},{"task_slug":"prompt-engineering","task_name":"Prompt Engineering"}],"methods":[{"method_slug":"diffusion","method_name":"Diffusion"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2407.01606","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2407.01606"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/ruocwang/dpo-diffusion","reach":null}],"summary":{"ran_draft_wrong":2,"unverified":1},"by_repo_kind":{"official":{"samples":3,"ran":2,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":3,"samples":[{"code_sha256_prefix":"36ae6024ae1ec9c5","entry":"get_t_schedule","repo":"ruocwang/dpo-diffusion","repo_kind":"official","path":"src/optimizers/gradient/gpo.py","file_url":"https://github.com/ruocwang/dpo-diffusion/blob/HEAD/src/optimizers/gradient/gpo.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"36ae6024ae1ec9c5"}},{"code_sha256_prefix":"acf8dd50c0490553","entry":"get_t_schedule","repo":"ruocwang/dpo-diffusion","repo_kind":"official","path":"src/optimizers/gradient/gpo.py","file_url":"https://github.com/ruocwang/dpo-diffusion/blob/HEAD/src/optimizers/gradient/gpo.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"acf8dd50c0490553"}},{"code_sha256_prefix":"2cf9785bc719b6b4","entry":"GradientPromptOptimizer","repo":"ruocwang/dpo-diffusion","repo_kind":"official","path":"src/optimizers/gradient/gpo.py","file_url":"https://github.com/ruocwang/dpo-diffusion/blob/HEAD/src/optimizers/gradient/gpo.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"2cf9785bc719b6b4"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}