{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/melting-point-mobile-evaluation-of-language","title":"MELTing point: Mobile Evaluation of Language Transformers","arxiv_id":"2403.12844","date":"2024-03-19","proceeding":null,"authors":["Stefanos Laskaridis","Kleomenis Katevas","Lorenzo Minto","Hamed Haddadi"],"abstract":"Transformers have revolutionized the machine learning landscape, gradually making their way into everyday tasks and equipping our computers with \"sparks of intelligence\". However, their runtime requirements have prevented them from being broadly deployed on mobile. As personal devices become increasingly powerful and prompt privacy becomes an ever more pressing issue, we explore the current state of mobile execution of Large Language Models (LLMs). To achieve this, we have created our own automation infrastructure, MELT, which supports the headless execution and benchmarking of LLMs on device, supporting different models, devices and frameworks, including Android, iOS and Nvidia Jetson devices. We evaluate popular instruction fine-tuned LLMs and leverage different frameworks to measure their end-to-end and granular performance, tracing their memory and energy requirements along the way. Our analysis is the first systematic study of on-device LLM execution, quantifying performance, energy efficiency and accuracy across various state-of-the-art models and showcases the state of on-device intelligence in the era of hyperscale models. Results highlight the performance heterogeneity across targets and corroborates that LLM inference is largely memory-bound. Quantization drastically reduces memory requirements and renders execution viable, but at a non-negligible accuracy cost. Drawing from its energy footprint and thermal behavior, the continuous execution of LLMs remains elusive, as both factors negatively affect user experience. Last, our experience shows that the ecosystem is still in its infancy, and algorithmic as well as hardware breakthroughs can significantly shift the execution cost. We expect NPU acceleration, and framework-hardware co-design to be the biggest bet towards efficient standalone execution, with the alternative of offloading tailored towards edge deployments.","url_abs":"https://arxiv.org/abs/2403.12844v4","url_pdf":"https://arxiv.org/pdf/2403.12844v4.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"melting-point-mobile-evaluation-of-language","repo_url":"https://github.com/brave-experiments/melt-public","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"none","reach":{"status":"ok","spdx":"NOASSERTION"}}],"tasks":[{"task_slug":"benchmarking","task_name":"Benchmarking"},{"task_slug":"quantization","task_name":"Quantization"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2403.12844","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2403.12844"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/brave-experiments/melt-public","reach":{"status":"ok","spdx":"NOASSERTION"}}],"summary":{"ran":7,"unverified":4},"by_repo_kind":{"official":{"samples":11,"ran":7,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":11,"samples":[{"code_sha256_prefix":"ebd488c33c6f542b","entry":"decode_quant_method","repo":"brave-experiments/melt-public","repo_kind":"official","path":"src/models/convert_utils/awq_utils.py","file_url":"https://github.com/brave-experiments/melt-public/blob/HEAD/src/models/convert_utils/awq_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"ebd488c33c6f542b"}},{"code_sha256_prefix":"d4d97ec85344adfa","entry":"llama_change_model_config_eos","repo":"brave-experiments/melt-public","repo_kind":"official","path":"src/models/convert_utils/llama_utils.py","file_url":"https://github.com/brave-experiments/melt-public/blob/HEAD/src/models/convert_utils/llama_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"d4d97ec85344adfa"}},{"code_sha256_prefix":"dc442bca6ccc09ba","entry":"merge_ops","repo":"brave-experiments/melt-public","repo_kind":"official","path":"src/parsers/utils/utils.py","file_url":"https://github.com/brave-experiments/melt-public/blob/HEAD/src/parsers/utils/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"dc442bca6ccc09ba"}},{"code_sha256_prefix":"dcd0cefbbd3a0fb9","entry":"mlc_get_max_length","repo":"brave-experiments/melt-public","repo_kind":"official","path":"src/models/convert_utils/mlc_utils.py","file_url":"https://github.com/brave-experiments/melt-public/blob/HEAD/src/models/convert_utils/mlc_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"dcd0cefbbd3a0fb9"}},{"code_sha256_prefix":"e25a568ffee9a3ca","entry":"mlc_translate_config_to_model_config","repo":"brave-experiments/melt-public","repo_kind":"official","path":"src/models/convert_utils/mlc_utils.py","file_url":"https://github.com/brave-experiments/melt-public/blob/HEAD/src/models/convert_utils/mlc_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"e25a568ffee9a3ca"}},{"code_sha256_prefix":"3c0cac5e5b160574","entry":"parse_json","repo":"brave-experiments/melt-public","repo_kind":"official","path":"src/parsers/utils/mlc_utils.py","file_url":"https://github.com/brave-experiments/melt-public/blob/HEAD/src/parsers/utils/mlc_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"3c0cac5e5b160574"}},{"code_sha256_prefix":"08965441176409fe","entry":"parse_logfile","repo":"brave-experiments/melt-public","repo_kind":"official","path":"src/parsers/utils/mlc_utils.py","file_url":"https://github.com/brave-experiments/melt-public/blob/HEAD/src/parsers/utils/mlc_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"08965441176409fe"}},{"code_sha256_prefix":"c682b93b401b16f5","entry":"convert_mlc","repo":"brave-experiments/melt-public","repo_kind":"official","path":"src/models/convert_utils/mlc_utils.py","file_url":"https://github.com/brave-experiments/melt-public/blob/HEAD/src/models/convert_utils/mlc_utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"c682b93b401b16f5"}},{"code_sha256_prefix":"6834dba45870141d","entry":"normalize_calls","repo":"brave-experiments/melt-public","repo_kind":"official","path":"src/parsers/utils/mlc_utils.py","file_url":"https://github.com/brave-experiments/melt-public/blob/HEAD/src/parsers/utils/mlc_utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"6834dba45870141d"}},{"code_sha256_prefix":"ef5a8d3df237c9ff","entry":"parse_file","repo":"brave-experiments/melt-public","repo_kind":"official","path":"src/parsers/utils/utils.py","file_url":"https://github.com/brave-experiments/melt-public/blob/HEAD/src/parsers/utils/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"ef5a8d3df237c9ff"}},{"code_sha256_prefix":"9d2469a55dd01a13","entry":"parse_logfile","repo":"brave-experiments/melt-public","repo_kind":"official","path":"src/parsers/utils/llamacpp_utils.py","file_url":"https://github.com/brave-experiments/melt-public/blob/HEAD/src/parsers/utils/llamacpp_utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"9d2469a55dd01a13"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}