{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/unsupervised-modality-adaptation-with-text-to","title":"Unsupervised Modality Adaptation with Text-to-Image Diffusion Models for Semantic Segmentation","arxiv_id":"2410.21708","date":"2024-10-29","proceeding":null,"authors":["Ruihao Xia","Yu Liang","Peng-Tao Jiang","Hao Zhang","Bo Li","Yang Tang","Pan Zhou"],"abstract":"Despite their success, unsupervised domain adaptation methods for semantic segmentation primarily focus on adaptation between image domains and do not utilize other abundant visual modalities like depth, infrared and event. This limitation hinders their performance and restricts their application in real-world multimodal scenarios. To address this issue, we propose Modality Adaptation with text-to-image Diffusion Models (MADM) for semantic segmentation task which utilizes text-to-image diffusion models pre-trained on extensive image-text pairs to enhance the model's cross-modality capabilities. Specifically, MADM comprises two key complementary components to tackle major challenges. First, due to the large modality gap, using one modal data to generate pseudo labels for another modality suffers from a significant drop in accuracy. To address this, MADM designs diffusion-based pseudo-label generation which adds latent noise to stabilize pseudo-labels and enhance label accuracy. Second, to overcome the limitations of latent low-resolution features in diffusion models, MADM introduces the label palette and latent regression which converts one-hot encoded labels into the RGB form by palette and regresses them in the latent space, thus ensuring the pre-trained decoder for up-sampling to obtain fine-grained features. Extensive experimental results demonstrate that MADM achieves state-of-the-art adaptation performance across various modality tasks, including images to depth, infrared, and event modalities. We open-source our code and models at https://github.com/XiaRho/MADM.","url_abs":"https://arxiv.org/abs/2410.21708v1","url_pdf":"https://arxiv.org/pdf/2410.21708v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"unsupervised-modality-adaptation-with-text-to","repo_url":"https://github.com/xiarho/madm","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":"domain-adaptation","task_name":"Domain Adaptation"},{"task_slug":"pseudo-label","task_name":"Pseudo Label"},{"task_slug":"semantic-segmentation","task_name":"Semantic Segmentation"},{"task_slug":"unsupervised-domain-adaptation","task_name":"Unsupervised Domain Adaptation"}],"methods":[{"method_slug":"diffusion","method_name":"Diffusion"},{"method_slug":"focus","method_name":"Focus"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2410.21708","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2410.21708"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/XiaRho/MADM","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/xiarho/madm","reach":{"status":"ok"}}],"summary":{"ran_fixture":1,"unverified":8},"by_repo_kind":{"official":{"samples":9,"ran":1,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":9,"samples":[{"code_sha256_prefix":"b2b9a1966482ef96","entry":"reduce_loss","repo":"XiaRho/MADM","repo_kind":"official","path":"modeling/criterion.py","file_url":"https://github.com/XiaRho/MADM/blob/HEAD/modeling/criterion.py","link_basis":"harvester_set","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"b2b9a1966482ef96"}},{"code_sha256_prefix":"208961850384d92d","entry":"auto_scale_workers","repo":"XiaRho/MADM","repo_kind":"official","path":"config/utils.py","file_url":"https://github.com/XiaRho/MADM/blob/HEAD/config/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"208961850384d92d"}},{"code_sha256_prefix":"69e60dbee942ddfa","entry":"batched_input_to_device","repo":"XiaRho/MADM","repo_kind":"official","path":"modeling/preprocess.py","file_url":"https://github.com/XiaRho/MADM/blob/HEAD/modeling/preprocess.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"69e60dbee942ddfa"}},{"code_sha256_prefix":"ef1831a6bdc512e6","entry":"get_dataset_from_loader","repo":"XiaRho/MADM","repo_kind":"official","path":"engine/defaults.py","file_url":"https://github.com/XiaRho/MADM/blob/HEAD/engine/defaults.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"ef1831a6bdc512e6"}},{"code_sha256_prefix":"c6bf0d68150e96b7","entry":"get_grad_norm","repo":"XiaRho/MADM","repo_kind":"official","path":"engine/train_loop.py","file_url":"https://github.com/XiaRho/MADM/blob/HEAD/engine/train_loop.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"c6bf0d68150e96b7"}},{"code_sha256_prefix":"b9e85d4f33e5f142","entry":"get_model_from_module","repo":"XiaRho/MADM","repo_kind":"official","path":"engine/defaults.py","file_url":"https://github.com/XiaRho/MADM/blob/HEAD/engine/defaults.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"b9e85d4f33e5f142"}},{"code_sha256_prefix":"e9eb46e7a17bab8c","entry":"get_optimizer_parameters","repo":"XiaRho/MADM","repo_kind":"official","path":"engine/train_loop.py","file_url":"https://github.com/XiaRho/MADM/blob/HEAD/engine/train_loop.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"e9eb46e7a17bab8c"}},{"code_sha256_prefix":"b3e569142043c972","entry":"group_by_prefix","repo":"XiaRho/MADM","repo_kind":"official","path":"checkpoint/odise_checkpointer.py","file_url":"https://github.com/XiaRho/MADM/blob/HEAD/checkpoint/odise_checkpointer.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"b3e569142043c972"}},{"code_sha256_prefix":"9a78b68ccf3be3bb","entry":"weight_reduce_loss","repo":"XiaRho/MADM","repo_kind":"official","path":"modeling/criterion.py","file_url":"https://github.com/XiaRho/MADM/blob/HEAD/modeling/criterion.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"9a78b68ccf3be3bb"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}