{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/beat-a-large-scale-semantic-and-emotional","title":"BEAT: A Large-Scale Semantic and Emotional Multi-Modal Dataset for Conversational Gestures Synthesis","arxiv_id":"2203.05297","date":"2022-03-10","proceeding":null,"authors":["Haiyang Liu","Zihao Zhu","Naoya Iwamoto","Yichen Peng","Zhengqing Li","You Zhou","Elif Bozkurt","Bo Zheng"],"abstract":"Achieving realistic, vivid, and human-like synthesized conversational gestures conditioned on multi-modal data is still an unsolved problem due to the lack of available datasets, models and standard evaluation metrics. To address this, we build Body-Expression-Audio-Text dataset, BEAT, which has i) 76 hours, high-quality, multi-modal data captured from 30 speakers talking with eight different emotions and in four different languages, ii) 32 millions frame-level emotion and semantic relevance annotations. Our statistical analysis on BEAT demonstrates the correlation of conversational gestures with facial expressions, emotions, and semantics, in addition to the known correlation with audio, text, and speaker identity. Based on this observation, we propose a baseline model, Cascaded Motion Network (CaMN), which consists of above six modalities modeled in a cascaded architecture for gesture synthesis. To evaluate the semantic relevancy, we introduce a metric, Semantic Relevance Gesture Recall (SRGR). Qualitative and quantitative experiments demonstrate metrics' validness, ground truth data quality, and baseline's state-of-the-art performance. To the best of our knowledge, BEAT is the largest motion capture dataset for investigating human gestures, which may contribute to a number of different research fields, including controllable gesture synthesis, cross-modality analysis, and emotional gesture recognition. The data, code and model are available on https://pantomatrix.github.io/BEAT/.","url_abs":"https://arxiv.org/abs/2203.05297v5","url_pdf":"https://arxiv.org/pdf/2203.05297v5.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"beat-a-large-scale-semantic-and-emotional","repo_url":"https://github.com/PantoMatrix/PantoMatrix","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"gesture-generation","task_name":"Gesture Generation"},{"task_slug":"gesture-recognition","task_name":"Gesture Recognition"}],"methods":[],"datasets_introduced":[{"slug":"beat","name":"BEAT","full_name":"Body-Expression-Audio-Text"}],"methods_introduced":[],"results":[{"leaderboard":"/sota/gesture-generation-on-beat","task":"Gesture Generation","dataset":"BEAT","model":"CaMN","rank_in_archive_order":1,"of":5,"metrics":{"FID":"122.8"},"uses_additional_data":false},{"leaderboard":"/sota/gesture-generation-on-beat2","task":"Gesture Generation","dataset":"BEAT2","model":"CaMN","rank_in_archive_order":10,"of":14,"metrics":{"FGD":"0.6644"},"uses_additional_data":false}],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2203.05297","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2203.05297"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/PantoMatrix/PantoMatrix","reach":null}],"summary":{"ran_fixture":2,"ran_honours":1},"by_repo_kind":{"official":{"samples":3,"ran":3,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":3,"samples":[{"code_sha256_prefix":"2c39643f3b2a0e35","entry":"matrix_to_axis_angle","repo":"PantoMatrix/PantoMatrix","repo_kind":"official","path":"models/camn_audio/modeling_camn_audio.py","file_url":"https://github.com/PantoMatrix/PantoMatrix/blob/HEAD/models/camn_audio/modeling_camn_audio.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"2c39643f3b2a0e35"}},{"code_sha256_prefix":"371530b9072dd179","entry":"matrix_to_quaternion","repo":"PantoMatrix/PantoMatrix","repo_kind":"official","path":"models/camn_audio/modeling_camn_audio.py","file_url":"https://github.com/PantoMatrix/PantoMatrix/blob/HEAD/models/camn_audio/modeling_camn_audio.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"371530b9072dd179"}},{"code_sha256_prefix":"1ff40bde986ccb61","entry":"quaternion_to_axis_angle","repo":"PantoMatrix/PantoMatrix","repo_kind":"official","path":"models/camn_audio/modeling_camn_audio.py","file_url":"https://github.com/PantoMatrix/PantoMatrix/blob/HEAD/models/camn_audio/modeling_camn_audio.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"1ff40bde986ccb61"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}