{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/harmonic-nas-hardware-aware-multimodal-neural","title":"Harmonic-NAS: Hardware-Aware Multimodal Neural Architecture Search on Resource-constrained Devices","arxiv_id":"2309.06612","date":"2023-09-12","proceeding":null,"authors":["Mohamed Imed Eddine Ghebriout","Halima Bouzidi","Smail Niar","Hamza Ouarnoughi"],"abstract":"The recent surge of interest surrounding Multimodal Neural Networks (MM-NN) is attributed to their ability to effectively process and integrate multiscale information from diverse data sources. MM-NNs extract and fuse features from multiple modalities using adequate unimodal backbones and specific fusion networks. Although this helps strengthen the multimodal information representation, designing such networks is labor-intensive. It requires tuning the architectural parameters of the unimodal backbones, choosing the fusing point, and selecting the operations for fusion. Furthermore, multimodality AI is emerging as a cutting-edge option in Internet of Things (IoT) systems where inference latency and energy consumption are critical metrics in addition to accuracy. In this paper, we propose Harmonic-NAS, a framework for the joint optimization of unimodal backbones and multimodal fusion networks with hardware awareness on resource-constrained devices. Harmonic-NAS involves a two-tier optimization approach for the unimodal backbone architectures and fusion strategy and operators. By incorporating the hardware dimension into the optimization, evaluation results on various devices and multimodal datasets have demonstrated the superiority of Harmonic-NAS over state-of-the-art approaches achieving up to 10.9% accuracy improvement, 1.91x latency reduction, and 2.14x energy efficiency gain.","url_abs":"https://arxiv.org/abs/2309.06612v2","url_pdf":"https://arxiv.org/pdf/2309.06612v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"harmonic-nas-hardware-aware-multimodal-neural","repo_url":"https://github.com/mohamed-imed-eddine/harmonic-nas","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":"classification","task_name":"General Classification"},{"task_slug":"multimodal-text-and-image-classification","task_name":"Multimodal Text and Image Classification"},{"task_slug":"architecture-search","task_name":"Neural Architecture Search"}],"methods":[{"method_slug":"neural-architecture-search","method_name":"Neural Architecture Search"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2309.06612","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2309.06612"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/mohamed-imed-eddine/harmonic-nas","reach":{"status":"ok"}}],"summary":{"unverified":2},"by_repo_kind":{"official":{"samples":2,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":2,"samples":[{"code_sha256_prefix":"bf8726d3804eafb1","entry":"CreateOp","repo":"mohamed-imed-eddine/harmonic-nas","repo_kind":"official","path":"fusion_search/auxiliary/aux_models.py","file_url":"https://github.com/mohamed-imed-eddine/harmonic-nas/blob/HEAD/fusion_search/auxiliary/aux_models.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"bf8726d3804eafb1"}},{"code_sha256_prefix":"73c29b0970500c05","entry":"genotype","repo":"mohamed-imed-eddine/harmonic-nas","repo_kind":"official","path":"fusion_search/genotype.py","file_url":"https://github.com/mohamed-imed-eddine/harmonic-nas/blob/HEAD/fusion_search/genotype.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"73c29b0970500c05"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}