{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/clay-a-controllable-large-scale-generative","title":"CLAY: A Controllable Large-scale Generative Model for Creating High-quality 3D Assets","arxiv_id":"2406.13897","date":"2024-05-30","proceeding":null,"authors":["Longwen Zhang","Ziyu Wang","Qixuan Zhang","QIwei Qiu","Anqi Pang","Haoran Jiang","Wei Yang","Lan Xu","Jingyi Yu"],"abstract":"In the realm of digital creativity, our potential to craft intricate 3D worlds from imagination is often hampered by the limitations of existing digital tools, which demand extensive expertise and efforts. To narrow this disparity, we introduce CLAY, a 3D geometry and material generator designed to effortlessly transform human imagination into intricate 3D digital structures. CLAY supports classic text or image inputs as well as 3D-aware controls from diverse primitives (multi-view images, voxels, bounding boxes, point clouds, implicit representations, etc). At its core is a large-scale generative model composed of a multi-resolution Variational Autoencoder (VAE) and a minimalistic latent Diffusion Transformer (DiT), to extract rich 3D priors directly from a diverse range of 3D geometries. Specifically, it adopts neural fields to represent continuous and complete surfaces and uses a geometry generative module with pure transformer blocks in latent space. We present a progressive training scheme to train CLAY on an ultra large 3D model dataset obtained through a carefully designed processing pipeline, resulting in a 3D native geometry generator with 1.5 billion parameters. For appearance generation, CLAY sets out to produce physically-based rendering (PBR) textures by employing a multi-view material diffusion model that can generate 2K resolution textures with diffuse, roughness, and metallic modalities. We demonstrate using CLAY for a range of controllable 3D asset creations, from sketchy conceptual designs to production ready assets with intricate details. Even first time users can easily use CLAY to bring their vivid 3D imaginations to life, unleashing unlimited creativity.","url_abs":"https://arxiv.org/abs/2406.13897v1","url_pdf":"https://arxiv.org/pdf/2406.13897v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"clay-a-controllable-large-scale-generative","repo_url":"https://github.com/Seed3D/Dora","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}},{"paper_slug":"clay-a-controllable-large-scale-generative","repo_url":"https://github.com/yixunliang/unitex","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"jax","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"2k","task_name":"2k"},{"task_slug":"3d-geometry","task_name":"3D geometry"}],"methods":[{"method_slug":"absolute-position-encodings","method_name":"Absolute Position Encodings"},{"method_slug":"adam","method_name":"Adam"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"bpe","method_name":"BPE"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"diffusion","method_name":"Diffusion"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"label-smoothing","method_name":"Label Smoothing"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"position-wise-feed-forward-layer","method_name":"Position-Wise Feed-Forward Layer"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"transformer","method_name":"Transformer"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2406.13897","atlas_url":"https://app.syntology.ai/?focus=2406.13897","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2406.13897"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/yixunliang/unitex","reach":{"status":"ok","spdx":"Apache-2.0"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/Seed3D/Dora","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran_honours":1,"ran":3,"unverified":4},"by_repo_kind":{"listed":{"samples":8,"ran":4,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"c56b7ef16f309a45","entry":"gaussian","repo":"yixunliang/unitex","repo_kind":"listed","path":"LTM/craftsman/utils/loss.py","file_url":"https://github.com/yixunliang/unitex/blob/HEAD/LTM/craftsman/utils/loss.py","link_basis":"harvester_set","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"c56b7ef16f309a45"}},{"code_sha256_prefix":"ac0e42d6fbcfbbe6","entry":"l1_loss","repo":"yixunliang/unitex","repo_kind":"listed","path":"LTM/craftsman/utils/loss.py","file_url":"https://github.com/yixunliang/unitex/blob/HEAD/LTM/craftsman/utils/loss.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"ac0e42d6fbcfbbe6"}},{"code_sha256_prefix":"8c3b0f873ba11813","entry":"l2_loss","repo":"yixunliang/unitex","repo_kind":"listed","path":"LTM/craftsman/utils/loss.py","file_url":"https://github.com/yixunliang/unitex/blob/HEAD/LTM/craftsman/utils/loss.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"8c3b0f873ba11813"}},{"code_sha256_prefix":"13812c4cff22d2f4","entry":"tile_sample","repo":"yixunliang/unitex","repo_kind":"listed","path":"TSD_SR/sr_pipeline.py","file_url":"https://github.com/yixunliang/unitex/blob/HEAD/TSD_SR/sr_pipeline.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"13812c4cff22d2f4"}},{"code_sha256_prefix":"8d3cd21cd5c3181a","entry":"AttentionBlock","repo":"yixunliang/unitex","repo_kind":"listed","path":"TSD_SR/basicsr/archs/dfdnet_util.py","file_url":"https://github.com/yixunliang/unitex/blob/HEAD/TSD_SR/basicsr/archs/dfdnet_util.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"8d3cd21cd5c3181a"}},{"code_sha256_prefix":"b7c6741eed724350","entry":"adaptive_instance_normalization","repo":"yixunliang/unitex","repo_kind":"listed","path":"TSD_SR/basicsr/archs/dfdnet_util.py","file_url":"https://github.com/yixunliang/unitex/blob/HEAD/TSD_SR/basicsr/archs/dfdnet_util.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"b7c6741eed724350"}},{"code_sha256_prefix":"5fe25138e6151526","entry":"build_ltm","repo":"yixunliang/unitex","repo_kind":"listed","path":"pipeline.py","file_url":"https://github.com/yixunliang/unitex/blob/HEAD/pipeline.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"5fe25138e6151526"}},{"code_sha256_prefix":"a7e8af4ef5efc131","entry":"calc_mean_std","repo":"yixunliang/unitex","repo_kind":"listed","path":"TSD_SR/basicsr/archs/dfdnet_util.py","file_url":"https://github.com/yixunliang/unitex/blob/HEAD/TSD_SR/basicsr/archs/dfdnet_util.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"a7e8af4ef5efc131"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}