{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/continuous-piecewise-affine-based-motion","title":"Continuous Piecewise-Affine Based Motion Model for Image Animation","arxiv_id":"2401.09146","date":"2024-01-17","proceeding":null,"authors":["Hexiang Wang","Fengqi Liu","Qianyu Zhou","Ran Yi","Xin Tan","Lizhuang Ma"],"abstract":"Image animation aims to bring static images to life according to driving videos and create engaging visual content that can be used for various purposes such as animation, entertainment, and education. Recent unsupervised methods utilize affine and thin-plate spline transformations based on keypoints to transfer the motion in driving frames to the source image. However, limited by the expressive power of the transformations used, these methods always produce poor results when the gap between the motion in the driving frame and the source image is large. To address this issue, we propose to model motion from the source image to the driving frame in highly-expressive diffeomorphism spaces. Firstly, we introduce Continuous Piecewise-Affine based (CPAB) transformation to model the motion and present a well-designed inference algorithm to generate CPAB transformation from control keypoints. Secondly, we propose a SAM-guided keypoint semantic loss to further constrain the keypoint extraction process and improve the semantic consistency between the corresponding keypoints on the source and driving images. Finally, we design a structure alignment loss to align the structure-related features extracted from driving and generated images, thus helping the generator generate results that are more consistent with the driving action. Extensive experiments on four datasets demonstrate the effectiveness of our method against state-of-the-art competitors quantitatively and qualitatively. Code will be publicly available at: https://github.com/DevilPG/AAAI2024-CPABMM.","url_abs":"https://arxiv.org/abs/2401.09146v1","url_pdf":"https://arxiv.org/pdf/2401.09146v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"continuous-piecewise-affine-based-motion","repo_url":"https://github.com/devilpg/aaai2024-cpabmm","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":"image-animation","task_name":"Image Animation"}],"methods":[{"method_slug":"align","method_name":"ALIGN"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2401.09146","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2401.09146"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/devilpg/aaai2024-cpabmm","reach":{"status":"ok"}},{"provenance":"deterministic:regex_extraction","url":"https://github.com/DevilPG/AAAI2024-CPABMM","reach":{"status":"ok"}}],"summary":{"ran_draft_wrong":1,"ran_fixture":2,"ran":2,"ran_honours":1,"unverified":1},"by_repo_kind":{"official":{"samples":7,"ran":6,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":7,"samples":[{"code_sha256_prefix":"e9f612ce15936ed2","entry":"attn_cosine_sim","repo":"DevilPG/AAAI2024-CPABMM","repo_kind":"official","path":"modules/vit_extractor.py","file_url":"https://github.com/DevilPG/AAAI2024-CPABMM/blob/HEAD/modules/vit_extractor.py","link_basis":"plan_row","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"e9f612ce15936ed2"}},{"code_sha256_prefix":"cba98ca3e7ba2298","entry":"kp2gaussian","repo":"DevilPG/AAAI2024-CPABMM","repo_kind":"official","path":"modules/util.py","file_url":"https://github.com/DevilPG/AAAI2024-CPABMM/blob/HEAD/modules/util.py","link_basis":"plan_row","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"cba98ca3e7ba2298"}},{"code_sha256_prefix":"fe3cbec3030f03b9","entry":"make_coordinate_grid","repo":"DevilPG/AAAI2024-CPABMM","repo_kind":"official","path":"modules/util.py","file_url":"https://github.com/DevilPG/AAAI2024-CPABMM/blob/HEAD/modules/util.py","link_basis":"plan_row","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"fe3cbec3030f03b9"}},{"code_sha256_prefix":"2d8aff37407bc6e7","entry":"random_scale","repo":"DevilPG/AAAI2024-CPABMM","repo_kind":"official","path":"train_avd.py","file_url":"https://github.com/DevilPG/AAAI2024-CPABMM/blob/HEAD/train_avd.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"2d8aff37407bc6e7"}},{"code_sha256_prefix":"a41523c0d5ff87ec","entry":"read_video","repo":"DevilPG/AAAI2024-CPABMM","repo_kind":"official","path":"frames_dataset.py","file_url":"https://github.com/DevilPG/AAAI2024-CPABMM/blob/HEAD/frames_dataset.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"a41523c0d5ff87ec"}},{"code_sha256_prefix":"c299e0261d0f079a","entry":"to_homogeneous","repo":"DevilPG/AAAI2024-CPABMM","repo_kind":"official","path":"modules/util.py","file_url":"https://github.com/DevilPG/AAAI2024-CPABMM/blob/HEAD/modules/util.py","link_basis":"plan_row","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"c299e0261d0f079a"}},{"code_sha256_prefix":"74ca7b86672df314","entry":"detach_kp","repo":"DevilPG/AAAI2024-CPABMM","repo_kind":"official","path":"modules/model.py","file_url":"https://github.com/DevilPG/AAAI2024-CPABMM/blob/HEAD/modules/model.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"74ca7b86672df314"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}