{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/markup-to-image-diffusion-models-with","title":"Markup-to-Image Diffusion Models with Scheduled Sampling","arxiv_id":"2210.05147","date":"2022-10-11","proceeding":null,"authors":["Yuntian Deng","Noriyuki Kojima","Alexander M. Rush"],"abstract":"Building on recent advances in image generation, we present a fully data-driven approach to rendering markup into images. The approach is based on diffusion models, which parameterize the distribution of data using a sequence of denoising operations on top of a Gaussian noise distribution. We view the diffusion denoising process as a sequential decision making process, and show that it exhibits compounding errors similar to exposure bias issues in imitation learning problems. To mitigate these issues, we adapt the scheduled sampling algorithm to diffusion training. We conduct experiments on four markup datasets: mathematical formulas (LaTeX), table layouts (HTML), sheet music (LilyPond), and molecular images (SMILES). These experiments each verify the effectiveness of the diffusion process and the use of scheduled sampling to fix generation issues. These results also show that the markup-to-image task presents a useful controlled compositional setting for diagnosing and analyzing generative image models.","url_abs":"https://arxiv.org/abs/2210.05147v1","url_pdf":"https://arxiv.org/pdf/2210.05147v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"markup-to-image-diffusion-models-with","repo_url":"https://github.com/da03/markup2im","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"decision-making","task_name":"Decision Making"},{"task_slug":"denoising","task_name":"Denoising"},{"task_slug":"image-generation","task_name":"Image Generation"},{"task_slug":"imitation-learning","task_name":"Imitation Learning"},{"task_slug":"sequential-decision-making","task_name":"Sequential Decision Making"}],"methods":[{"method_slug":"diffusion","method_name":"Diffusion"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2210.05147","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2210.05147"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/da03/markup2im","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"unverified":8},"by_repo_kind":{"official":{"samples":8,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"c18ca7f68e0b6c5b","entry":"clip_score","repo":"da03/markup2im","repo_kind":"official","path":"eval_utils/clip_utils.py","file_url":"https://github.com/da03/markup2im/blob/HEAD/eval_utils/clip_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c18ca7f68e0b6c5b"}},{"code_sha256_prefix":"c02507c005d13ce7","entry":"encode_text","repo":"da03/markup2im","repo_kind":"official","path":"src/markup2im_models.py","file_url":"https://github.com/da03/markup2im/blob/HEAD/src/markup2im_models.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c02507c005d13ce7"}},{"code_sha256_prefix":"0c42525b641a7c61","entry":"get_column_features","repo":"da03/markup2im","repo_kind":"official","path":"eval_utils/metric_utils.py","file_url":"https://github.com/da03/markup2im/blob/HEAD/eval_utils/metric_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"0c42525b641a7c61"}},{"code_sha256_prefix":"0ac64afe7da2ea47","entry":"get_encoder_model_type","repo":"da03/markup2im","repo_kind":"official","path":"src/markup2im_constants.py","file_url":"https://github.com/da03/markup2im/blob/HEAD/src/markup2im_constants.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"0ac64afe7da2ea47"}},{"code_sha256_prefix":"c4c8ed0cccf40206","entry":"get_image_size","repo":"da03/markup2im","repo_kind":"official","path":"src/markup2im_constants.py","file_url":"https://github.com/da03/markup2im/blob/HEAD/src/markup2im_constants.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c4c8ed0cccf40206"}},{"code_sha256_prefix":"9ed74ba3f1a0f8f4","entry":"get_input_field","repo":"da03/markup2im","repo_kind":"official","path":"src/markup2im_constants.py","file_url":"https://github.com/da03/markup2im/blob/HEAD/src/markup2im_constants.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"9ed74ba3f1a0f8f4"}},{"code_sha256_prefix":"dd2266bcddfb50b0","entry":"load_image_cv2","repo":"da03/markup2im","repo_kind":"official","path":"eval_utils/metric_utils.py","file_url":"https://github.com/da03/markup2im/blob/HEAD/eval_utils/metric_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"dd2266bcddfb50b0"}},{"code_sha256_prefix":"cd8beb90d5c8da82","entry":"load_image_pil","repo":"da03/markup2im","repo_kind":"official","path":"eval_utils/metric_utils.py","file_url":"https://github.com/da03/markup2im/blob/HEAD/eval_utils/metric_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"cd8beb90d5c8da82"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}