{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/words-can-shift-dynamically-adjusting-word","title":"Words Can Shift: Dynamically Adjusting Word Representations Using Nonverbal Behaviors","arxiv_id":"1811.09362","date":"2018-11-23","proceeding":null,"authors":["Yansen Wang","Ying Shen","Zhun Liu","Paul Pu Liang","Amir Zadeh","Louis-Philippe Morency"],"abstract":"Humans convey their intentions through the usage of both verbal and nonverbal\nbehaviors during face-to-face communication. Speaker intentions often vary\ndynamically depending on different nonverbal contexts, such as vocal patterns\nand facial expressions. As a result, when modeling human language, it is\nessential to not only consider the literal meaning of the words but also the\nnonverbal contexts in which these words appear. To better model human language,\nwe first model expressive nonverbal representations by analyzing the\nfine-grained visual and acoustic patterns that occur during word segments. In\naddition, we seek to capture the dynamic nature of nonverbal intents by\nshifting word representations based on the accompanying nonverbal behaviors. To\nthis end, we propose the Recurrent Attended Variation Embedding Network (RAVEN)\nthat models the fine-grained structure of nonverbal subword sequences and\ndynamically shifts word representations based on nonverbal cues. Our proposed\nmodel achieves competitive performance on two publicly available datasets for\nmultimodal sentiment analysis and emotion recognition. We also visualize the\nshifted word representations in different nonverbal contexts and summarize\ncommon patterns regarding multimodal variations of word representations.","url_abs":"http://arxiv.org/abs/1811.09362v2","url_pdf":"http://arxiv.org/pdf/1811.09362v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"words-can-shift-dynamically-adjusting-word","repo_url":"https://github.com/victorywys/RAVEN","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok"}},{"paper_slug":"words-can-shift-dynamically-adjusting-word","repo_url":"https://github.com/cmu-multicomp-lab/cmu-multimodalsdk","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"words-can-shift-dynamically-adjusting-word","repo_url":"https://github.com/maochf/CMU-MultimodalSDK","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"NOASSERTION"}},{"paper_slug":"words-can-shift-dynamically-adjusting-word","repo_url":"https://github.com/matsuolab/cmu-multimodalsdk","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"NOASSERTION"}},{"paper_slug":"words-can-shift-dynamically-adjusting-word","repo_url":"https://github.com/righ120/multimodal_nlp","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok"}}],"tasks":[{"task_slug":"emotion-recognition","task_name":"Emotion Recognition"},{"task_slug":"multimodal-sentiment-analysis","task_name":"Multimodal Sentiment Analysis"},{"task_slug":"sentiment-analysis","task_name":"Sentiment Analysis"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":null,"atlas_url":"https://app.syntology.ai/?focus=1811.09362","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"1811.09362"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/righ120/multimodal_nlp","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/cmu-multicomp-lab/cmu-multimodalsdk","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/maochf/CMU-MultimodalSDK","reach":{"status":"ok","spdx":"NOASSERTION"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/matsuolab/cmu-multimodalsdk","reach":{"status":"ok","spdx":"NOASSERTION"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/victorywys/RAVEN","reach":{"status":"ok"}}],"summary":{"unverified":2},"by_repo_kind":{"listed":{"samples":2,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"99c31f0713c8bf9a","entry":"progress_bar","repo":"cmu-multicomp-lab/cmu-multimodalsdk","repo_kind":"listed","path":"mmsdk/mmdatasdk/log/log.py","file_url":"https://github.com/cmu-multicomp-lab/cmu-multimodalsdk/blob/HEAD/mmsdk/mmdatasdk/log/log.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"99c31f0713c8bf9a"}},{"code_sha256_prefix":"3039fa7760ccae96","entry":"status","repo":"cmu-multicomp-lab/cmu-multimodalsdk","repo_kind":"listed","path":"mmsdk/mmdatasdk/log/log.py","file_url":"https://github.com/cmu-multicomp-lab/cmu-multimodalsdk/blob/HEAD/mmsdk/mmdatasdk/log/log.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"3039fa7760ccae96"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}