{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/sketchdreamer-interactive-text-augmented","title":"SketchDreamer: Interactive Text-Augmented Creative Sketch Ideation","arxiv_id":"2308.14191","date":"2023-08-27","proceeding":null,"authors":["Zhiyu Qu","Tao Xiang","Yi-Zhe Song"],"abstract":"Artificial Intelligence Generated Content (AIGC) has shown remarkable progress in generating realistic images. However, in this paper, we take a step \"backward\" and address AIGC for the most rudimentary visual modality of human sketches. Our objective is on the creative nature of sketches, and that creative sketching should take the form of an interactive process. We further enable text to drive the sketch ideation process, allowing creativity to be freely defined, while simultaneously tackling the challenge of \"I can't sketch\". We present a method to generate controlled sketches using a text-conditioned diffusion model trained on pixel representations of images. Our proposed approach, referred to as SketchDreamer, integrates a differentiable rasteriser of Bezier curves that optimises an initial input to distil abstract semantic knowledge from a pretrained diffusion model. We utilise Score Distillation Sampling to learn a sketch that aligns with a given caption, which importantly enable both text and sketch to interact with the ideation process. Our objective is to empower non-professional users to create sketches and, through a series of optimisation processes, transform a narrative into a storyboard by expanding the text prompt while making minor adjustments to the sketch input. Through this work, we hope to aspire the way we create visual content, democratise the creative process, and inspire further research in enhancing human creativity in AIGC. The code is available at \\url{https://github.com/WinKawaks/SketchDreamer}.","url_abs":"https://arxiv.org/abs/2308.14191v1","url_pdf":"https://arxiv.org/pdf/2308.14191v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"sketchdreamer-interactive-text-augmented","repo_url":"https://github.com/winkawaks/sketchdreamer","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[],"methods":[{"method_slug":"diffusion","method_name":"Diffusion"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2308.14191","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2308.14191"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/winkawaks/sketchdreamer","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran":8,"ran_violates":1,"unverified":1},"by_repo_kind":{"official":{"samples":10,"ran":9,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"66f70c7814190f31","entry":"convert_to_grayscale","repo":"winkawaks/sketchdreamer","repo_kind":"official","path":"painterly_rendering.py","file_url":"https://github.com/winkawaks/sketchdreamer/blob/HEAD/painterly_rendering.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"66f70c7814190f31"}},{"code_sha256_prefix":"44940c05f6245895","entry":"cos_layers","repo":"winkawaks/sketchdreamer","repo_kind":"official","path":"models/loss.py","file_url":"https://github.com/winkawaks/sketchdreamer/blob/HEAD/models/loss.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"44940c05f6245895"}},{"code_sha256_prefix":"2a729f02c74a7ad0","entry":"fix_image_scale","repo":"winkawaks/sketchdreamer","repo_kind":"official","path":"sketch_utils.py","file_url":"https://github.com/winkawaks/sketchdreamer/blob/HEAD/sketch_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"2a729f02c74a7ad0"}},{"code_sha256_prefix":"188456eee49a9baf","entry":"gradCAM","repo":"winkawaks/sketchdreamer","repo_kind":"official","path":"models/painter_params.py","file_url":"https://github.com/winkawaks/sketchdreamer/blob/HEAD/models/painter_params.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"188456eee49a9baf"}},{"code_sha256_prefix":"5a90146b8021ad22","entry":"interpret","repo":"winkawaks/sketchdreamer","repo_kind":"official","path":"models/painter_params.py","file_url":"https://github.com/winkawaks/sketchdreamer/blob/HEAD/models/painter_params.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"5a90146b8021ad22"}},{"code_sha256_prefix":"b904dcd060391f9c","entry":"l1_layers","repo":"winkawaks/sketchdreamer","repo_kind":"official","path":"models/loss.py","file_url":"https://github.com/winkawaks/sketchdreamer/blob/HEAD/models/loss.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"b904dcd060391f9c"}},{"code_sha256_prefix":"3aa37678e56ebdb0","entry":"l2_layers","repo":"winkawaks/sketchdreamer","repo_kind":"official","path":"models/loss.py","file_url":"https://github.com/winkawaks/sketchdreamer/blob/HEAD/models/loss.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"3aa37678e56ebdb0"}},{"code_sha256_prefix":"bc3ccaf78ecd6394","entry":"multi_head_attention_forward","repo":"winkawaks/sketchdreamer","repo_kind":"official","path":"CLIP_/clip/auxilary.py","file_url":"https://github.com/winkawaks/sketchdreamer/blob/HEAD/CLIP_/clip/auxilary.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"bc3ccaf78ecd6394"}},{"code_sha256_prefix":"10ebc4ca56bae3c2","entry":"str_to_bool","repo":"winkawaks/sketchdreamer","repo_kind":"official","path":"config.py","file_url":"https://github.com/winkawaks/sketchdreamer/blob/HEAD/config.py","link_basis":"plan_row","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"10ebc4ca56bae3c2"}},{"code_sha256_prefix":"7ecaa1ab867d461d","entry":"build_model","repo":"winkawaks/sketchdreamer","repo_kind":"official","path":"CLIP_/clip/model.py","file_url":"https://github.com/winkawaks/sketchdreamer/blob/HEAD/CLIP_/clip/model.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"7ecaa1ab867d461d"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}