{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/from-text-to-pose-to-image-improving","title":"From Text to Pose to Image: Improving Diffusion Model Control and Quality","arxiv_id":"2411.12872","date":"2024-11-19","proceeding":null,"authors":["Clément Bonnet","Ariel N. Lee","Franck Wertel","Antoine Tamano","Tanguy Cizain","Pablo Ducru"],"abstract":"In the last two years, text-to-image diffusion models have become extremely popular. As their quality and usage increase, a major concern has been the need for better output control. In addition to prompt engineering, one effective method to improve the controllability of diffusion models has been to condition them on additional modalities such as image style, depth map, or keypoints. This forms the basis of ControlNets or Adapters. When attempting to apply these methods to control human poses in outputs of text-to-image diffusion models, two main challenges have arisen. The first challenge is generating poses following a wide range of semantic text descriptions, for which previous methods involved searching for a pose within a dataset of (caption, pose) pairs. The second challenge is conditioning image generation on a specified pose while keeping both high aesthetic and high pose fidelity. In this article, we fix these two main issues by introducing a text-to-pose (T2P) generative model alongside a new sampling algorithm, and a new pose adapter that incorporates more pose keypoints for higher pose fidelity. Together, these two new state-of-the-art models enable, for the first time, a generative text-to-pose-to-image framework for higher pose control in diffusion models. We release all models and the code used for the experiments at https://github.com/clement-bonnet/text-to-pose.","url_abs":"https://arxiv.org/abs/2411.12872v2","url_pdf":"https://arxiv.org/pdf/2411.12872v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"from-text-to-pose-to-image-improving","repo_url":"https://github.com/clement-bonnet/text-to-pose","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"image-generation","task_name":"Image Generation"},{"task_slug":"prompt-engineering","task_name":"Prompt Engineering"}],"methods":[{"method_slug":"adapter","method_name":"Adapter"},{"method_slug":"diffusion","method_name":"Diffusion"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":null,"atlas_url":"https://app.syntology.ai/?focus=2411.12872","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2411.12872"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/clement-bonnet/text-to-pose","reach":null}],"summary":{"ran":5,"ran_honours":2,"ran_draft_wrong":1,"unverified":3},"by_repo_kind":{"official":{"samples":11,"ran":8,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"023c96154955c6a3","entry":"CausalSelfAttention","repo":"clement-bonnet/text-to-pose","repo_kind":"official","path":"t2p/model.py","file_url":"https://github.com/clement-bonnet/text-to-pose/blob/HEAD/t2p/model.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"023c96154955c6a3"}},{"code_sha256_prefix":"0ba7b9be8f372eca","entry":"PoseBodiesDict","repo":"clement-bonnet/text-to-pose","repo_kind":"official","path":"t2p/model.py","file_url":"https://github.com/clement-bonnet/text-to-pose/blob/HEAD/t2p/model.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"0ba7b9be8f372eca"}},{"code_sha256_prefix":"f2654aebe02ac7c2","entry":"PoseDict","repo":"clement-bonnet/text-to-pose","repo_kind":"official","path":"t2p/model.py","file_url":"https://github.com/clement-bonnet/text-to-pose/blob/HEAD/t2p/model.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f2654aebe02ac7c2"}},{"code_sha256_prefix":"a29bd54e5b52c32f","entry":"T2PSample","repo":"clement-bonnet/text-to-pose","repo_kind":"official","path":"t2p/model.py","file_url":"https://github.com/clement-bonnet/text-to-pose/blob/HEAD/t2p/model.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a29bd54e5b52c32f"}},{"code_sha256_prefix":"17aeb94103ec6da9","entry":"TransformerLayer","repo":"clement-bonnet/text-to-pose","repo_kind":"official","path":"t2p/model.py","file_url":"https://github.com/clement-bonnet/text-to-pose/blob/HEAD/t2p/model.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"17aeb94103ec6da9"}},{"code_sha256_prefix":"0f906a50a669c77e","entry":"_to_device","repo":"clement-bonnet/text-to-pose","repo_kind":"official","path":"t2p/model.py","file_url":"https://github.com/clement-bonnet/text-to-pose/blob/HEAD/t2p/model.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"0f906a50a669c77e"}},{"code_sha256_prefix":"4622271331855d00","entry":"_to_numpy","repo":"clement-bonnet/text-to-pose","repo_kind":"official","path":"t2p/model.py","file_url":"https://github.com/clement-bonnet/text-to-pose/blob/HEAD/t2p/model.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"4622271331855d00"}},{"code_sha256_prefix":"0d31a62464b7a4b6","entry":"_unsqueeze","repo":"clement-bonnet/text-to-pose","repo_kind":"official","path":"t2p/model.py","file_url":"https://github.com/clement-bonnet/text-to-pose/blob/HEAD/t2p/model.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"0d31a62464b7a4b6"}},{"code_sha256_prefix":"f245bd7aee3e82c1","entry":"T2PTransformer","repo":"clement-bonnet/text-to-pose","repo_kind":"official","path":"t2p/model.py","file_url":"https://github.com/clement-bonnet/text-to-pose/blob/HEAD/t2p/model.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f245bd7aee3e82c1"}},{"code_sha256_prefix":"3772a310fcc48bc7","entry":"tempered_sampling","repo":"clement-bonnet/text-to-pose","repo_kind":"official","path":"t2p/sampling.py","file_url":"https://github.com/clement-bonnet/text-to-pose/blob/HEAD/t2p/sampling.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"3772a310fcc48bc7"}},{"code_sha256_prefix":"363591a993ed83b2","entry":"tempered_sampling","repo":"clement-bonnet/text-to-pose","repo_kind":"official","path":"t2p/model.py","file_url":"https://github.com/clement-bonnet/text-to-pose/blob/HEAD/t2p/model.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"363591a993ed83b2"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}