{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/analyzing-the-hardware-software-implications","title":"MMBench: Benchmarking End-to-End Multi-modal DNNs and Understanding Their Hardware-Software Implications","arxiv_id":"2212.01241","date":"2022-12-02","proceeding":null,"authors":["Cheng Xu","Xiaofeng Hou","Jiacheng Liu","Chao Li","Tianhao Huang","Xiaozhi Zhu","Mo Niu","Lingyu Sun","Peng Tang","Tongqiao Xu","Kwang-Ting Cheng","Minyi Guo"],"abstract":"The explosive growth of various types of big data and advances in AI technologies have catalyzed a new type of workloads called multi-modal DNNs. Multi-modal DNNs are capable of interpreting and reasoning about information from multiple modalities, making them more applicable to real-world AI scenarios. In recent research, multi-modal DNNs have outperformed the best uni-modal DNN in a wide range of distributed computing applications from traditional multimedia systems to emerging autonomous edge systems. However, despite their importance and superiority, very limited research attention has been devoted to understand the characteristics of multi-modal DNNs and their implications on current computing software/hardware platforms. Existing benchmarks either target uni-modal DNNs or only focus on the algorithm characteristics of multi-modal DNNs. There lacks representative benchmark suites that provide comprehensive system and architecture level analysis of multi-modal networks. To advance the understanding of these multi-modal DNN workloads and facilitate related research, we present MMBench, an open-source, end-to-end benchmark suite consisting of a set of real-world multi-modal DNN workloads with relevant performance metrics for evaluation. We then use MMBench to conduct an in-depth analysis on the characteristics of multi-modal DNNs. We demonstrate their unique characteristics of clear multi-stage execution, frequent synchronization and high heterogeneity, which distinguish them from conventional uni-modal DNNs. Finally, we conduct a case study and extend our benchmark to edge devices. We hope that our work can provide insights for future software/hardware design and optimization to underpin multi-modal DNNs on both cloud and edge computing platforms.","url_abs":"https://arxiv.org/abs/2212.01241v4","url_pdf":"https://arxiv.org/pdf/2212.01241v4.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"links_only","authors_date_abstract":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license), from the Kaggle arXiv metadata snapshot of 2026-09-12"},"code_links":[{"paper_slug":"analyzing-the-hardware-software-implications","repo_url":"https://github.com/xfhelen/mmbench","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2212.01241","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2212.01241"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/xfhelen/mmbench","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":4,"ran_draft_wrong":1,"unverified":6},"by_repo_kind":{"official":{"samples":11,"ran":5,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"809e3aa9964cb3ea","entry":"AUPRC","repo":"xfhelen/mmbench","repo_kind":"official","path":"models/eval_scripts/performance.py","file_url":"https://github.com/xfhelen/mmbench/blob/HEAD/models/eval_scripts/performance.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"809e3aa9964cb3ea"}},{"code_sha256_prefix":"191c253b532edecf","entry":"f1_score","repo":"xfhelen/mmbench","repo_kind":"official","path":"models/eval_scripts/performance.py","file_url":"https://github.com/xfhelen/mmbench/blob/HEAD/models/eval_scripts/performance.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"191c253b532edecf"}},{"code_sha256_prefix":"f86744b3991d473e","entry":"getallparams","repo":"xfhelen/mmbench","repo_kind":"official","path":"models/eval_scripts/complexity.py","file_url":"https://github.com/xfhelen/mmbench/blob/HEAD/models/eval_scripts/complexity.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f86744b3991d473e"}},{"code_sha256_prefix":"376f134d99d8ed75","entry":"normalization","repo":"xfhelen/mmbench","repo_kind":"official","path":"applications/Medical-Segmentation/layers.py","file_url":"https://github.com/xfhelen/mmbench/blob/HEAD/applications/Medical-Segmentation/layers.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"376f134d99d8ed75"}},{"code_sha256_prefix":"c93aad3618c35efa","entry":"ptsort","repo":"xfhelen/mmbench","repo_kind":"official","path":"models/eval_scripts/performance.py","file_url":"https://github.com/xfhelen/mmbench/blob/HEAD/models/eval_scripts/performance.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c93aad3618c35efa"}},{"code_sha256_prefix":"f2fd4bd22929f069","entry":"create_film_layer","repo":"xfhelen/mmbench","repo_kind":"official","path":"models/fusions/common_fusions.py","file_url":"https://github.com/xfhelen/mmbench/blob/HEAD/models/fusions/common_fusions.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f2fd4bd22929f069"}},{"code_sha256_prefix":"a146f39b2da4131e","entry":"create_film_layer_v2","repo":"xfhelen/mmbench","repo_kind":"official","path":"models/fusions/common_fusions.py","file_url":"https://github.com/xfhelen/mmbench/blob/HEAD/models/fusions/common_fusions.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a146f39b2da4131e"}},{"code_sha256_prefix":"21907c83b0724472","entry":"get_possible_layer_configurations","repo":"xfhelen/mmbench","repo_kind":"official","path":"models/fusions/searchable.py","file_url":"https://github.com/xfhelen/mmbench/blob/HEAD/models/fusions/searchable.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"21907c83b0724472"}},{"code_sha256_prefix":"3483e2fd20eb7606","entry":"nosigmloss1d","repo":"xfhelen/mmbench","repo_kind":"official","path":"models/objective_functions/recon.py","file_url":"https://github.com/xfhelen/mmbench/blob/HEAD/models/objective_functions/recon.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"3483e2fd20eb7606"}},{"code_sha256_prefix":"84ca4fef1e15264f","entry":"sigmloss1d","repo":"xfhelen/mmbench","repo_kind":"official","path":"models/objective_functions/recon.py","file_url":"https://github.com/xfhelen/mmbench/blob/HEAD/models/objective_functions/recon.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"84ca4fef1e15264f"}},{"code_sha256_prefix":"ad19dbde74fdba65","entry":"sigmloss1dcentercrop","repo":"xfhelen/mmbench","repo_kind":"official","path":"models/objective_functions/recon.py","file_url":"https://github.com/xfhelen/mmbench/blob/HEAD/models/objective_functions/recon.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ad19dbde74fdba65"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}