{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/exploring-discrete-diffusion-models-for-image","title":"Exploring Discrete Diffusion Models for Image Captioning","arxiv_id":"2211.11694","date":"2022-11-21","proceeding":null,"authors":["Zixin Zhu","Yixuan Wei","JianFeng Wang","Zhe Gan","Zheng Zhang","Le Wang","Gang Hua","Lijuan Wang","Zicheng Liu","Han Hu"],"abstract":"The image captioning task is typically realized by an auto-regressive method that decodes the text tokens one by one. We present a diffusion-based captioning model, dubbed the name DDCap, to allow more decoding flexibility. Unlike image generation, where the output is continuous and redundant with a fixed length, texts in image captions are categorical and short with varied lengths. Therefore, naively applying the discrete diffusion model to text decoding does not work well, as shown in our experiments. To address the performance gap, we propose several key techniques including best-first inference, concentrated attention mask, text length prediction, and image-free training. On COCO without additional caption pre-training, it achieves a CIDEr score of 117.8, which is +5.0 higher than the auto-regressive baseline with the same architecture in the controlled setting. It also performs +26.8 higher CIDEr score than the auto-regressive baseline (230.3 v.s.203.5) on a caption infilling task. With 4M vision-language pre-training images and the base-sized model, we reach a CIDEr score of 125.1 on COCO, which is competitive to the best well-developed auto-regressive frameworks. The code is available at https://github.com/buxiangzhiren/DDCap.","url_abs":"https://arxiv.org/abs/2211.11694v2","url_pdf":"https://arxiv.org/pdf/2211.11694v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"exploring-discrete-diffusion-models-for-image","repo_url":"https://github.com/buxiangzhiren/ddcap","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"image-captioning","task_name":"Image Captioning"},{"task_slug":"image-generation","task_name":"Image Generation"}],"methods":[{"method_slug":"diffusion","method_name":"Diffusion"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2211.11694","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2211.11694"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/buxiangzhiren/ddcap","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":2,"ran_honours":2,"unverified":14},"by_repo_kind":{"official":{"samples":18,"ran":4,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"e33b112514a9e4b0","entry":"generate2","repo":"buxiangzhiren/ddcap","repo_kind":"official","path":"predict.py","file_url":"https://github.com/buxiangzhiren/ddcap/blob/HEAD/predict.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e33b112514a9e4b0"}},{"code_sha256_prefix":"883320eb62f15d12","entry":"generate_beam","repo":"buxiangzhiren/ddcap","repo_kind":"official","path":"predict.py","file_url":"https://github.com/buxiangzhiren/ddcap/blob/HEAD/predict.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"883320eb62f15d12"}},{"code_sha256_prefix":"479771b269fc5883","entry":"log_1_min_a","repo":"buxiangzhiren/ddcap","repo_kind":"official","path":"train_tclip.py","file_url":"https://github.com/buxiangzhiren/ddcap/blob/HEAD/train_tclip.py","link_basis":"harvester_set","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"479771b269fc5883"}},{"code_sha256_prefix":"032c0dece6594020","entry":"log_add_exp","repo":"buxiangzhiren/ddcap","repo_kind":"official","path":"train_tclip.py","file_url":"https://github.com/buxiangzhiren/ddcap/blob/HEAD/train_tclip.py","link_basis":"harvester_set","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"032c0dece6594020"}},{"code_sha256_prefix":"c68cb2401825d6a1","entry":"add_period","repo":"buxiangzhiren/ddcap","repo_kind":"official","path":"parse_conceptual.py","file_url":"https://github.com/buxiangzhiren/ddcap/blob/HEAD/parse_conceptual.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c68cb2401825d6a1"}},{"code_sha256_prefix":"8b17971afcbd7a4f","entry":"build_model","repo":"buxiangzhiren/ddcap","repo_kind":"official","path":"clip1/model.py","file_url":"https://github.com/buxiangzhiren/ddcap/blob/HEAD/clip1/model.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"8b17971afcbd7a4f"}},{"code_sha256_prefix":"5e666c43b8dba5d1","entry":"build_scheduler","repo":"buxiangzhiren/ddcap","repo_kind":"official","path":"lr_scheduler.py","file_url":"https://github.com/buxiangzhiren/ddcap/blob/HEAD/lr_scheduler.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"5e666c43b8dba5d1"}},{"code_sha256_prefix":"5a97e7f261c6118d","entry":"check_imports","repo":"buxiangzhiren/ddcap","repo_kind":"official","path":"dynamic_module_utils.py","file_url":"https://github.com/buxiangzhiren/ddcap/blob/HEAD/dynamic_module_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"5a97e7f261c6118d"}},{"code_sha256_prefix":"09c8479d9a5b3e06","entry":"extract","repo":"buxiangzhiren/ddcap","repo_kind":"official","path":"train_tclip.py","file_url":"https://github.com/buxiangzhiren/ddcap/blob/HEAD/train_tclip.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"09c8479d9a5b3e06"}},{"code_sha256_prefix":"adea8f61592be69f","entry":"find_pruneable_heads_and_indices","repo":"buxiangzhiren/ddcap","repo_kind":"official","path":"fix_pre.py","file_url":"https://github.com/buxiangzhiren/ddcap/blob/HEAD/fix_pre.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"adea8f61592be69f"}},{"code_sha256_prefix":"0a286fb23ba005b9","entry":"get_image","repo":"buxiangzhiren/ddcap","repo_kind":"official","path":"parse_conceptual.py","file_url":"https://github.com/buxiangzhiren/ddcap/blob/HEAD/parse_conceptual.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"0a286fb23ba005b9"}},{"code_sha256_prefix":"7d86410784062a90","entry":"get_model_parameters_info","repo":"buxiangzhiren/ddcap","repo_kind":"official","path":"misc.py","file_url":"https://github.com/buxiangzhiren/ddcap/blob/HEAD/misc.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"7d86410784062a90"}},{"code_sha256_prefix":"373405b421a7ccd1","entry":"get_relative_import_files","repo":"buxiangzhiren/ddcap","repo_kind":"official","path":"dynamic_module_utils.py","file_url":"https://github.com/buxiangzhiren/ddcap/blob/HEAD/dynamic_module_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"373405b421a7ccd1"}},{"code_sha256_prefix":"9f0143385b707e8a","entry":"get_relative_imports","repo":"buxiangzhiren/ddcap","repo_kind":"official","path":"dynamic_module_utils.py","file_url":"https://github.com/buxiangzhiren/ddcap/blob/HEAD/dynamic_module_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"9f0143385b707e8a"}},{"code_sha256_prefix":"cb4e99dd4b50f69d","entry":"load_tf_weights_in_gpt2","repo":"buxiangzhiren/ddcap","repo_kind":"official","path":"tf_adpt.py","file_url":"https://github.com/buxiangzhiren/ddcap/blob/HEAD/tf_adpt.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"cb4e99dd4b50f69d"}},{"code_sha256_prefix":"66531358d7cefe44","entry":"merge_opts_to_config","repo":"buxiangzhiren/ddcap","repo_kind":"official","path":"misc.py","file_url":"https://github.com/buxiangzhiren/ddcap/blob/HEAD/misc.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"66531358d7cefe44"}},{"code_sha256_prefix":"647f36aefa59b564","entry":"modify_config_for_debug","repo":"buxiangzhiren/ddcap","repo_kind":"official","path":"misc.py","file_url":"https://github.com/buxiangzhiren/ddcap/blob/HEAD/misc.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"647f36aefa59b564"}},{"code_sha256_prefix":"50449115c68a0ddb","entry":"thread","repo":"buxiangzhiren/ddcap","repo_kind":"official","path":"parse_conceptual.py","file_url":"https://github.com/buxiangzhiren/ddcap/blob/HEAD/parse_conceptual.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"50449115c68a0ddb"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}