{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/altdiffusion-a-multilingual-text-to-image","title":"AltDiffusion: A Multilingual Text-to-Image Diffusion Model","arxiv_id":"2308.09991","date":"2023-08-19","proceeding":null,"authors":["Fulong Ye","Guang Liu","Xinya Wu","Ledell Wu"],"abstract":"Large Text-to-Image(T2I) diffusion models have shown a remarkable capability to produce photorealistic and diverse images based on text inputs. However, existing works only support limited language input, e.g., English, Chinese, and Japanese, leaving users beyond these languages underserved and blocking the global expansion of T2I models. Therefore, this paper presents AltDiffusion, a novel multilingual T2I diffusion model that supports eighteen different languages. Specifically, we first train a multilingual text encoder based on the knowledge distillation. Then we plug it into a pretrained English-only diffusion model and train the model with a two-stage schema to enhance the multilingual capability, including concept alignment and quality improvement stage on a large-scale multilingual dataset. Furthermore, we introduce a new benchmark, which includes Multilingual-General-18(MG-18) and Multilingual-Cultural-18(MC-18) datasets, to evaluate the capabilities of T2I diffusion models for generating high-quality images and capturing culture-specific concepts in different languages. Experimental results on both MG-18 and MC-18 demonstrate that AltDiffusion outperforms current state-of-the-art T2I models, e.g., Stable Diffusion in multilingual understanding, especially with respect to culture-specific concepts, while still having comparable capability for generating high-quality images. All source code and checkpoints could be found in https://github.com/superhero-7/AltDiffuson.","url_abs":"https://arxiv.org/abs/2308.09991v2","url_pdf":"https://arxiv.org/pdf/2308.09991v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"altdiffusion-a-multilingual-text-to-image","repo_url":"https://github.com/superhero-7/altdiffuson","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":"blocking","task_name":"Blocking"},{"task_slug":"concept-alignment","task_name":"Concept Alignment"},{"task_slug":"knowledge-distillation","task_name":"Knowledge Distillation"},{"task_slug":"model","task_name":"model"}],"methods":[{"method_slug":"altclip","method_name":"AltCLIP"},{"method_slug":"altdiffusion","method_name":"AltDiffusion"},{"method_slug":"diffusion","method_name":"Diffusion"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2308.09991","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2308.09991"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/superhero-7/altdiffuson","reach":{"status":"ok"}}],"summary":{"ran_draft_wrong":1,"ran":6,"ran_violates":2,"unverified":2},"by_repo_kind":{"official":{"samples":11,"ran":9,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":11,"samples":[{"code_sha256_prefix":"fe5dd5258046898c","entry":"always","repo":"superhero-7/altdiffuson","repo_kind":"official","path":"src/lm/x_transformer.py","file_url":"https://github.com/superhero-7/altdiffuson/blob/HEAD/src/lm/x_transformer.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"fe5dd5258046898c"}},{"code_sha256_prefix":"24639f6e7d9a9755","entry":"append_data","repo":"superhero-7/altdiffuson","repo_kind":"official","path":"misc/coco_evaluation/generate.py","file_url":"https://github.com/superhero-7/altdiffuson/blob/HEAD/misc/coco_evaluation/generate.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"24639f6e7d9a9755"}},{"code_sha256_prefix":"91d8ba10bf7f3893","entry":"append_data","repo":"superhero-7/altdiffuson","repo_kind":"official","path":"misc/coco_evaluation/generate_taiyi.py","file_url":"https://github.com/superhero-7/altdiffuson/blob/HEAD/misc/coco_evaluation/generate_taiyi.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"91d8ba10bf7f3893"}},{"code_sha256_prefix":"c1295c485299a09f","entry":"append_data","repo":"superhero-7/altdiffuson","repo_kind":"official","path":"misc/evaluation/generate.py","file_url":"https://github.com/superhero-7/altdiffuson/blob/HEAD/misc/evaluation/generate.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"c1295c485299a09f"}},{"code_sha256_prefix":"eb0622d39c08c6eb","entry":"count_samples","repo":"superhero-7/altdiffuson","repo_kind":"official","path":"src/data.py","file_url":"https://github.com/superhero-7/altdiffuson/blob/HEAD/src/data.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"eb0622d39c08c6eb"}},{"code_sha256_prefix":"424012cb37b31172","entry":"default","repo":"superhero-7/altdiffuson","repo_kind":"official","path":"src/lm/x_transformer.py","file_url":"https://github.com/superhero-7/altdiffuson/blob/HEAD/src/lm/x_transformer.py","link_basis":"harvester_set","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"424012cb37b31172"}},{"code_sha256_prefix":"aa5486a3650902d8","entry":"exists","repo":"superhero-7/altdiffuson","repo_kind":"official","path":"src/lm/x_transformer.py","file_url":"https://github.com/superhero-7/altdiffuson/blob/HEAD/src/lm/x_transformer.py","link_basis":"harvester_set","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"aa5486a3650902d8"}},{"code_sha256_prefix":"b1368330cf0f5642","entry":"isimage","repo":"superhero-7/altdiffuson","repo_kind":"official","path":"src/ldm/util.py","file_url":"https://github.com/superhero-7/altdiffuson/blob/HEAD/src/ldm/util.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"b1368330cf0f5642"}},{"code_sha256_prefix":"d72762b700feee6f","entry":"ismap","repo":"superhero-7/altdiffuson","repo_kind":"official","path":"src/ldm/util.py","file_url":"https://github.com/superhero-7/altdiffuson/blob/HEAD/src/ldm/util.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"d72762b700feee6f"}},{"code_sha256_prefix":"f9bd2e83191afad1","entry":"log_txt_as_img","repo":"superhero-7/altdiffuson","repo_kind":"official","path":"src/ldm/util.py","file_url":"https://github.com/superhero-7/altdiffuson/blob/HEAD/src/ldm/util.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"f9bd2e83191afad1"}},{"code_sha256_prefix":"05df488890c8b9e2","entry":"preprocess_txt","repo":"superhero-7/altdiffuson","repo_kind":"official","path":"src/data.py","file_url":"https://github.com/superhero-7/altdiffuson/blob/HEAD/src/data.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"05df488890c8b9e2"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}