{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/canonicalfusion-generating-drivable-3d-human","title":"CanonicalFusion: Generating Drivable 3D Human Avatars from Multiple Images","arxiv_id":"2407.04345","date":"2024-07-05","proceeding":null,"authors":["Jisu Shin","Junmyeong Lee","Seongmin Lee","Min-Gyu Park","Ju-Mi Kang","Ju Hong Yoon","Hae-Gon Jeon"],"abstract":"We present a novel framework for reconstructing animatable human avatars from multiple images, termed CanonicalFusion. Our central concept involves integrating individual reconstruction results into the canonical space. To be specific, we first predict Linear Blend Skinning (LBS) weight maps and depth maps using a shared-encoder-dual-decoder network, enabling direct canonicalization of the 3D mesh from the predicted depth maps. Here, instead of predicting high-dimensional skinning weights, we infer compressed skinning weights, i.e., 3-dimensional vector, with the aid of pre-trained MLP networks. We also introduce a forward skinning-based differentiable rendering scheme to merge the reconstructed results from multiple images. This scheme refines the initial mesh by reposing the canonical mesh via the forward skinning and by minimizing photometric and geometric errors between the rendered and the predicted results. Our optimization scheme considers the position and color of vertices as well as the joint angles for each image, thereby mitigating the negative effects of pose errors. We conduct extensive experiments to demonstrate the effectiveness of our method and compare our CanonicalFusion with state-of-the-art methods. Our source codes are available at https://github.com/jsshin98/CanonicalFusion.","url_abs":"https://arxiv.org/abs/2407.04345v2","url_pdf":"https://arxiv.org/pdf/2407.04345v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"canonicalfusion-generating-drivable-3d-human","repo_url":"https://github.com/jsshin98/canonicalfusion","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"decoder","task_name":"Decoder"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2407.04345","atlas_url":"https://app.syntology.ai/?focus=2407.04345","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2407.04345"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/jsshin98/CanonicalFusion","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":3,"ran_honours":1,"unverified":2},"by_repo_kind":{"official":{"samples":6,"ran":4,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"0a589b69402596de","entry":"angle_prior","repo":"jsshin98/CanonicalFusion","repo_kind":"official","path":"utils/losses.py","file_url":"https://github.com/jsshin98/CanonicalFusion/blob/HEAD/utils/losses.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"0a589b69402596de"}},{"code_sha256_prefix":"f6b944f50d3f15ae","entry":"count_parameters","repo":"jsshin98/CanonicalFusion","repo_kind":"official","path":"reconstructor/train_color2model.py","file_url":"https://github.com/jsshin98/CanonicalFusion/blob/HEAD/reconstructor/train_color2model.py","link_basis":"harvester_set","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f6b944f50d3f15ae"}},{"code_sha256_prefix":"ac55fcb98792c961","entry":"gmof","repo":"jsshin98/CanonicalFusion","repo_kind":"official","path":"utils/losses.py","file_url":"https://github.com/jsshin98/CanonicalFusion/blob/HEAD/utils/losses.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ac55fcb98792c961"}},{"code_sha256_prefix":"752ef5eedf81dab6","entry":"init_semantic_labels","repo":"jsshin98/CanonicalFusion","repo_kind":"official","path":"apps/data_loader.py","file_url":"https://github.com/jsshin98/CanonicalFusion/blob/HEAD/apps/data_loader.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"752ef5eedf81dab6"}},{"code_sha256_prefix":"529920c432b75fee","entry":"init_variables","repo":"jsshin98/CanonicalFusion","repo_kind":"official","path":"reconstructor/train_color2model.py","file_url":"https://github.com/jsshin98/CanonicalFusion/blob/HEAD/reconstructor/train_color2model.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"529920c432b75fee"}},{"code_sha256_prefix":"bc70e3c7eb7c1354","entry":"train","repo":"jsshin98/CanonicalFusion","repo_kind":"official","path":"reconstructor/train_color2model.py","file_url":"https://github.com/jsshin98/CanonicalFusion/blob/HEAD/reconstructor/train_color2model.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"bc70e3c7eb7c1354"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}