{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/learned-representation-guided-diffusion","title":"Learned representation-guided diffusion models for large-image generation","arxiv_id":"2312.07330","date":"2023-12-12","proceeding":"CVPR 2024 1","authors":["Alexandros Graikos","Srikar Yellapragada","Minh-Quan Le","Saarthak Kapse","Prateek Prasanna","Joel Saltz","Dimitris Samaras"],"abstract":"To synthesize high-fidelity samples, diffusion models typically require auxiliary data to guide the generation process. However, it is impractical to procure the painstaking patch-level annotation effort required in specialized domains like histopathology and satellite imagery; it is often performed by domain experts and involves hundreds of millions of patches. Modern-day self-supervised learning (SSL) representations encode rich semantic and visual information. In this paper, we posit that such representations are expressive enough to act as proxies to fine-grained human labels. We introduce a novel approach that trains diffusion models conditioned on embeddings from SSL. Our diffusion models successfully project these features back to high-quality histopathology and remote sensing images. In addition, we construct larger images by assembling spatially consistent patches inferred from SSL embeddings, preserving long-range dependencies. Augmenting real data by generating variations of real images improves downstream classifier accuracy for patch-level and larger, image-scale classification tasks. Our models are effective even on datasets not encountered during training, demonstrating their robustness and generalizability. Generating images from learned embeddings is agnostic to the source of the embeddings. The SSL embeddings used to generate a large image can either be extracted from a reference image, or sampled from an auxiliary model conditioned on any related modality (e.g. class labels, text, genomic data). As proof of concept, we introduce the text-to-large image synthesis paradigm where we successfully synthesize large pathology and satellite images out of text descriptions.","url_abs":"https://arxiv.org/abs/2312.07330v2","url_pdf":"https://arxiv.org/pdf/2312.07330v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"learned-representation-guided-diffusion","repo_url":"https://github.com/cvlab-stonybrook/large-image-diffusion","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":"image-generation","task_name":"Image Generation"},{"task_slug":"self-supervised-learning","task_name":"Self-Supervised Learning"}],"methods":[{"method_slug":"diffusion","method_name":"Diffusion"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2312.07330","atlas_url":"https://app.syntology.ai/?focus=2312.07330","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2312.07330"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/cvlab-stonybrook/large-image-diffusion","reach":{"status":"ok"}}],"summary":{"ran":10,"ran_violates":1,"ran_draft_wrong":2,"unverified":1},"by_repo_kind":{"official":{"samples":14,"ran":13,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":14,"samples":[{"code_sha256_prefix":"b04722a19845dc90","entry":"analytic_kernel","repo":"cvlab-stonybrook/large-image-diffusion","repo_kind":"official","path":"ldm/modules/image_degradation/bsrgan.py","file_url":"https://github.com/cvlab-stonybrook/large-image-diffusion/blob/HEAD/ldm/modules/image_degradation/bsrgan.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"b04722a19845dc90"}},{"code_sha256_prefix":"5a0ca1bba88bb4b0","entry":"anisotropic_Gaussian","repo":"cvlab-stonybrook/large-image-diffusion","repo_kind":"official","path":"ldm/modules/image_degradation/bsrgan.py","file_url":"https://github.com/cvlab-stonybrook/large-image-diffusion/blob/HEAD/ldm/modules/image_degradation/bsrgan.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"5a0ca1bba88bb4b0"}},{"code_sha256_prefix":"082e51fd548d78b2","entry":"append_dims","repo":"cvlab-stonybrook/large-image-diffusion","repo_kind":"official","path":"ldm/models/diffusion/sampling_util.py","file_url":"https://github.com/cvlab-stonybrook/large-image-diffusion/blob/HEAD/ldm/models/diffusion/sampling_util.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"082e51fd548d78b2"}},{"code_sha256_prefix":"4cb732f513d69dfd","entry":"disabled_train","repo":"cvlab-stonybrook/large-image-diffusion","repo_kind":"official","path":"ldm/models/diffusion/ddpm.py","file_url":"https://github.com/cvlab-stonybrook/large-image-diffusion/blob/HEAD/ldm/models/diffusion/ddpm.py","link_basis":"harvester_set","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"4cb732f513d69dfd"}},{"code_sha256_prefix":"5bda79d6c0eb0ae5","entry":"expand_dims","repo":"cvlab-stonybrook/large-image-diffusion","repo_kind":"official","path":"ldm/models/diffusion/dpm_solver/dpm_solver.py","file_url":"https://github.com/cvlab-stonybrook/large-image-diffusion/blob/HEAD/ldm/models/diffusion/dpm_solver/dpm_solver.py","link_basis":"plan_row","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"5bda79d6c0eb0ae5"}},{"code_sha256_prefix":"f9d4c247328f3337","entry":"get_image_paths","repo":"cvlab-stonybrook/large-image-diffusion","repo_kind":"official","path":"ldm/modules/image_degradation/utils_image.py","file_url":"https://github.com/cvlab-stonybrook/large-image-diffusion/blob/HEAD/ldm/modules/image_degradation/utils_image.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"f9d4c247328f3337"}},{"code_sha256_prefix":"4099155bf06052b1","entry":"interpolate_fn","repo":"cvlab-stonybrook/large-image-diffusion","repo_kind":"official","path":"ldm/models/diffusion/dpm_solver/dpm_solver.py","file_url":"https://github.com/cvlab-stonybrook/large-image-diffusion/blob/HEAD/ldm/models/diffusion/dpm_solver/dpm_solver.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"4099155bf06052b1"}},{"code_sha256_prefix":"226f6afcd57ff476","entry":"is_image_file","repo":"cvlab-stonybrook/large-image-diffusion","repo_kind":"official","path":"ldm/modules/image_degradation/utils_image.py","file_url":"https://github.com/cvlab-stonybrook/large-image-diffusion/blob/HEAD/ldm/modules/image_degradation/utils_image.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"226f6afcd57ff476"}},{"code_sha256_prefix":"04df7487a791a8a1","entry":"modcrop_np","repo":"cvlab-stonybrook/large-image-diffusion","repo_kind":"official","path":"ldm/modules/image_degradation/bsrgan.py","file_url":"https://github.com/cvlab-stonybrook/large-image-diffusion/blob/HEAD/ldm/modules/image_degradation/bsrgan.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"04df7487a791a8a1"}},{"code_sha256_prefix":"689a4db9cb169a33","entry":"model_wrapper","repo":"cvlab-stonybrook/large-image-diffusion","repo_kind":"official","path":"ldm/models/diffusion/dpm_solver/dpm_solver.py","file_url":"https://github.com/cvlab-stonybrook/large-image-diffusion/blob/HEAD/ldm/models/diffusion/dpm_solver/dpm_solver.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"689a4db9cb169a33"}},{"code_sha256_prefix":"9898fe8b041f354c","entry":"norm_thresholding","repo":"cvlab-stonybrook/large-image-diffusion","repo_kind":"official","path":"ldm/models/diffusion/sampling_util.py","file_url":"https://github.com/cvlab-stonybrook/large-image-diffusion/blob/HEAD/ldm/models/diffusion/sampling_util.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"9898fe8b041f354c"}},{"code_sha256_prefix":"d86694c13b94ae3a","entry":"spatial_norm_thresholding","repo":"cvlab-stonybrook/large-image-diffusion","repo_kind":"official","path":"ldm/models/diffusion/sampling_util.py","file_url":"https://github.com/cvlab-stonybrook/large-image-diffusion/blob/HEAD/ldm/models/diffusion/sampling_util.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"d86694c13b94ae3a"}},{"code_sha256_prefix":"d48d8354986e3b0e","entry":"uniform_on_device","repo":"cvlab-stonybrook/large-image-diffusion","repo_kind":"official","path":"ldm/models/diffusion/ddpm.py","file_url":"https://github.com/cvlab-stonybrook/large-image-diffusion/blob/HEAD/ldm/models/diffusion/ddpm.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"d48d8354986e3b0e"}},{"code_sha256_prefix":"3984b3ec31e90ef0","entry":"patches_from_image","repo":"cvlab-stonybrook/large-image-diffusion","repo_kind":"official","path":"ldm/modules/image_degradation/utils_image.py","file_url":"https://github.com/cvlab-stonybrook/large-image-diffusion/blob/HEAD/ldm/modules/image_degradation/utils_image.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"3984b3ec31e90ef0"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}