{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/190600295","title":"Multimodal Transformer for Unaligned Multimodal Language Sequences","arxiv_id":"1906.00295","date":"2019-06-01","proceeding":"ACL 2019 7","authors":["Yao-Hung Hubert Tsai","Shaojie Bai","Paul Pu Liang","J. Zico Kolter","Louis-Philippe Morency","Ruslan Salakhutdinov"],"abstract":"Human language is often multimodal, which comprehends a mixture of natural language, facial gestures, and acoustic behaviors. However, two major challenges in modeling such multimodal human language time-series data exist: 1) inherent data non-alignment due to variable sampling rates for the sequences from each modality; and 2) long-range dependencies between elements across modalities. In this paper, we introduce the Multimodal Transformer (MulT) to generically address the above issues in an end-to-end manner without explicitly aligning the data. At the heart of our model is the directional pairwise crossmodal attention, which attends to interactions between multimodal sequences across distinct time steps and latently adapt streams from one modality to another. Comprehensive experiments on both aligned and non-aligned multimodal time-series show that our model outperforms state-of-the-art methods by a large margin. In addition, empirical analysis suggests that correlated crossmodal signals are able to be captured by the proposed crossmodal attention mechanism in MulT.","url_abs":"https://arxiv.org/abs/1906.00295v1","url_pdf":"https://arxiv.org/pdf/1906.00295v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"190600295","repo_url":"https://github.com/yaohungt/Multimodal-Transformer","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"190600295","repo_url":"https://github.com/JhnLee/multimodal-transformer","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}},{"paper_slug":"190600295","repo_url":"https://github.com/kenford953/graphcage","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"190600295","repo_url":"https://github.com/pliang279/MFN","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"multimodal-sentiment-analysis","task_name":"Multimodal Sentiment Analysis"},{"task_slug":"time-series-1","task_name":"Time Series"},{"task_slug":"time-series","task_name":"Time Series Analysis"}],"methods":[{"method_slug":"absolute-position-encodings","method_name":"Absolute Position Encodings"},{"method_slug":"adam","method_name":"Adam"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"bpe","method_name":"BPE"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"label-smoothing","method_name":"Label Smoothing"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"position-wise-feed-forward-layer","method_name":"Position-Wise Feed-Forward Layer"},{"method_slug":"relu","method_name":"ReLU"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"transformer","method_name":"Transformer"}],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/multimodal-sentiment-analysis-on-mosi","task":"Multimodal Sentiment Analysis","dataset":"MOSI","model":"MulT","rank_in_archive_order":6,"of":11,"metrics":{"Accuracy":"83","F1 score":"82.8"},"uses_additional_data":false}],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=1906.00295","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"1906.00295"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/JhnLee/multimodal-transformer","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/yaohungt/Multimodal-Transformer","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/kenford953/graphcage","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/pliang279/MFN","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran_draft_wrong":2,"ran":1,"ran_violates":1,"ran_honours":3,"ran_fixture":1,"unverified":9},"by_repo_kind":{"official":{"samples":7,"ran":2,"repositories":1},"listed":{"samples":8,"ran":4,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":10,"samples":[{"code_sha256_prefix":"02408cb7d7b045d1","entry":"LayerNorm","repo":null,"repo_kind":null,"path":null,"file_url":null,"link_basis":"identical_code_first_harvested_elsewhere","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":null,"inline_ok":false,"mcp_get_code":{"code_sha256":"02408cb7d7b045d1"}},{"code_sha256_prefix":"10865bbb99140edd","entry":"Linear","repo":null,"repo_kind":null,"path":null,"file_url":null,"link_basis":"identical_code_first_harvested_elsewhere","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":null,"inline_ok":false,"mcp_get_code":{"code_sha256":"10865bbb99140edd"}},{"code_sha256_prefix":"0056ce87dc2bc16f","entry":"SinusoidalPositionalEmbedding","repo":"kenford953/graphcage","repo_kind":"listed","path":"src/CrossmodalTransformer.py","file_url":"https://github.com/kenford953/graphcage/blob/HEAD/src/CrossmodalTransformer.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"0056ce87dc2bc16f"}},{"code_sha256_prefix":"184a5eddedaf6a42","entry":"buffered_future_mask","repo":"kenford953/graphcage","repo_kind":"listed","path":"src/CrossmodalTransformer.py","file_url":"https://github.com/kenford953/graphcage/blob/HEAD/src/CrossmodalTransformer.py","link_basis":"first_harvest_node","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"184a5eddedaf6a42"}},{"code_sha256_prefix":"8f266b9d616de9b9","entry":"fill_with_neg_inf","repo":"kenford953/graphcage","repo_kind":"listed","path":"src/CrossmodalTransformer.py","file_url":"https://github.com/kenford953/graphcage/blob/HEAD/src/CrossmodalTransformer.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"8f266b9d616de9b9"}},{"code_sha256_prefix":"21958d540bd87a8c","entry":"make_positions","repo":"kenford953/graphcage","repo_kind":"listed","path":"src/CrossmodalTransformer.py","file_url":"https://github.com/kenford953/graphcage/blob/HEAD/src/CrossmodalTransformer.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"21958d540bd87a8c"}},{"code_sha256_prefix":"9487efb0962df60b","entry":"multiclass_acc","repo":"yaohungt/Multimodal-Transformer","repo_kind":"official","path":"src/eval_metrics.py","file_url":"https://github.com/yaohungt/Multimodal-Transformer/blob/HEAD/src/eval_metrics.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"9487efb0962df60b"}},{"code_sha256_prefix":"6d4da8365bb0f5d1","entry":"weighted_accuracy","repo":"yaohungt/Multimodal-Transformer","repo_kind":"official","path":"src/eval_metrics.py","file_url":"https://github.com/yaohungt/Multimodal-Transformer/blob/HEAD/src/eval_metrics.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"6d4da8365bb0f5d1"}},{"code_sha256_prefix":"4e132df3b6dfa3c6","entry":"MULTModel","repo":"kenford953/graphcage","repo_kind":"listed","path":"src/CrossmodalTransformer.py","file_url":"https://github.com/kenford953/graphcage/blob/HEAD/src/CrossmodalTransformer.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"4e132df3b6dfa3c6"}},{"code_sha256_prefix":"81f5a0e8677cf0a6","entry":"MULTModel","repo":"yaohungt/Multimodal-Transformer","repo_kind":"official","path":"src/models.py","file_url":"https://github.com/yaohungt/Multimodal-Transformer/blob/HEAD/src/models.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"81f5a0e8677cf0a6"}},{"code_sha256_prefix":"34e57a8ff4c50a35","entry":"MultiheadAttention","repo":"kenford953/graphcage","repo_kind":"listed","path":"src/CrossmodalTransformer.py","file_url":"https://github.com/kenford953/graphcage/blob/HEAD/src/CrossmodalTransformer.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"34e57a8ff4c50a35"}},{"code_sha256_prefix":"a1bae24ff466241b","entry":"MultiheadAttention","repo":"yaohungt/Multimodal-Transformer","repo_kind":"official","path":"src/models.py","file_url":"https://github.com/yaohungt/Multimodal-Transformer/blob/HEAD/src/models.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a1bae24ff466241b"}},{"code_sha256_prefix":"da29e003f8b8fe95","entry":"TransformerEncoder","repo":"kenford953/graphcage","repo_kind":"listed","path":"src/CrossmodalTransformer.py","file_url":"https://github.com/kenford953/graphcage/blob/HEAD/src/CrossmodalTransformer.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"da29e003f8b8fe95"}},{"code_sha256_prefix":"8a709f2a4a8e7ede","entry":"TransformerEncoder","repo":"yaohungt/Multimodal-Transformer","repo_kind":"official","path":"src/models.py","file_url":"https://github.com/yaohungt/Multimodal-Transformer/blob/HEAD/src/models.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"8a709f2a4a8e7ede"}},{"code_sha256_prefix":"b04b6b2fd212ab1a","entry":"TransformerEncoderLayer","repo":"kenford953/graphcage","repo_kind":"listed","path":"src/CrossmodalTransformer.py","file_url":"https://github.com/kenford953/graphcage/blob/HEAD/src/CrossmodalTransformer.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"b04b6b2fd212ab1a"}},{"code_sha256_prefix":"9518e860e7fd09b2","entry":"TransformerEncoderLayer","repo":"yaohungt/Multimodal-Transformer","repo_kind":"official","path":"src/models.py","file_url":"https://github.com/yaohungt/Multimodal-Transformer/blob/HEAD/src/models.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"9518e860e7fd09b2"}},{"code_sha256_prefix":"52ec3727261ea9b7","entry":"eval_mosi","repo":"yaohungt/Multimodal-Transformer","repo_kind":"official","path":"src/eval_metrics.py","file_url":"https://github.com/yaohungt/Multimodal-Transformer/blob/HEAD/src/eval_metrics.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"52ec3727261ea9b7"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}