{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/mlperf-tiny-benchmark","title":"MLPerf Tiny Benchmark","arxiv_id":"2106.07597","date":"2021-06-14","proceeding":null,"authors":["Colby Banbury","Vijay Janapa Reddi","Peter Torelli","Jeremy Holleman","Nat Jeffries","Csaba Kiraly","Pietro Montino","David Kanter","Sebastian Ahmed","Danilo Pau","Urmish Thakker","Antonio Torrini","Peter Warden","Jay Cordaro","Giuseppe Di Guglielmo","Javier Duarte","Stephen Gibellini","Videet Parekh","Honson Tran","Nhan Tran","Niu Wenxu","Xu Xuesong"],"abstract":"Advancements in ultra-low-power tiny machine learning (TinyML) systems promise to unlock an entirely new class of smart applications. However, continued progress is limited by the lack of a widely accepted and easily reproducible benchmark for these systems. To meet this need, we present MLPerf Tiny, the first industry-standard benchmark suite for ultra-low-power tiny machine learning systems. The benchmark suite is the collaborative effort of more than 50 organizations from industry and academia and reflects the needs of the community. MLPerf Tiny measures the accuracy, latency, and energy of machine learning inference to properly evaluate the tradeoffs between systems. Additionally, MLPerf Tiny implements a modular design that enables benchmark submitters to show the benefits of their product, regardless of where it falls on the ML deployment stack, in a fair and reproducible manner. The suite features four benchmarks: keyword spotting, visual wake words, image classification, and anomaly detection.","url_abs":"https://arxiv.org/abs/2106.07597v4","url_pdf":"https://arxiv.org/pdf/2106.07597v4.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"mlperf-tiny-benchmark","repo_url":"https://github.com/mlcommons/tiny","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"tf","reach":{"status":"ok","spdx":"Apache-2.0"}},{"paper_slug":"mlperf-tiny-benchmark","repo_url":"https://github.com/mlcommons/tiny_results_v0.5","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"none","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"anomaly-detection","task_name":"Anomaly Detection"},{"task_slug":"machine-learning","task_name":"BIG-bench Machine Learning"},{"task_slug":"image-classification","task_name":"Image Classification"},{"task_slug":"keyword-spotting","task_name":"Keyword Spotting"},{"task_slug":"image-classification","task_name":"image-classification"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2106.07597","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2106.07597"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/mlcommons/tiny_results_v0.5","reach":{"status":"ok","spdx":"Apache-2.0"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/mlcommons/tiny","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran_violates":1,"unverified":15},"by_repo_kind":{"official":{"samples":16,"ran":1,"repositories":2}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"7c508037b40522af","entry":"str2bool","repo":"mlcommons/tiny_results_v0.5","repo_kind":"official","path":"closed/PengChengLaboratory/code/ad/ConvertModel.py","file_url":"https://github.com/mlcommons/tiny_results_v0.5/blob/HEAD/closed/PengChengLaboratory/code/ad/ConvertModel.py","link_basis":"harvester_set","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"7c508037b40522af"}},{"code_sha256_prefix":"7340ad06fb362e40","entry":"array_from_strings","repo":"mlcommons/tiny","repo_kind":"official","path":"benchmark/runner/streaming_ww_utils.py","file_url":"https://github.com/mlcommons/tiny/blob/HEAD/benchmark/runner/streaming_ww_utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"7340ad06fb362e40"}},{"code_sha256_prefix":"85ee9507d23e68c0","entry":"effnet_v2s","repo":"mlcommons/tiny","repo_kind":"official","path":"benchmark/training/image_classification/keras_model.py","file_url":"https://github.com/mlcommons/tiny/blob/HEAD/benchmark/training/image_classification/keras_model.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"85ee9507d23e68c0"}},{"code_sha256_prefix":"24bf521db881643a","entry":"get_baud_rate","repo":"mlcommons/tiny","repo_kind":"official","path":"benchmark/runner/runner_utils.py","file_url":"https://github.com/mlcommons/tiny/blob/HEAD/benchmark/runner/runner_utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"24bf521db881643a"}},{"code_sha256_prefix":"69600599471ffc0c","entry":"get_model","repo":"mlcommons/tiny","repo_kind":"official","path":"benchmark/training/anomaly_detection/keras_model.py","file_url":"https://github.com/mlcommons/tiny/blob/HEAD/benchmark/training/anomaly_detection/keras_model.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"69600599471ffc0c"}},{"code_sha256_prefix":"c3a0293fcad9180a","entry":"get_model","repo":"mlcommons/tiny_results_v0.5","repo_kind":"official","path":"closed/PengChengLaboratory/code/ad/tiny/v0.1/training/keyword_spotting/keras_model.py","file_url":"https://github.com/mlcommons/tiny_results_v0.5/blob/HEAD/closed/PengChengLaboratory/code/ad/tiny/v0.1/training/keyword_spotting/keras_model.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"c3a0293fcad9180a"}},{"code_sha256_prefix":"52eb372fa50a6b62","entry":"get_name","repo":"mlcommons/tiny","repo_kind":"official","path":"benchmark/runner/stream_wav_uart.py","file_url":"https://github.com/mlcommons/tiny/blob/HEAD/benchmark/runner/stream_wav_uart.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"52eb372fa50a6b62"}},{"code_sha256_prefix":"ffecdb9ac0a5c52f","entry":"get_response","repo":"mlcommons/tiny","repo_kind":"official","path":"benchmark/runner/stream_wav_uart.py","file_url":"https://github.com/mlcommons/tiny/blob/HEAD/benchmark/runner/stream_wav_uart.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"ffecdb9ac0a5c52f"}},{"code_sha256_prefix":"62ef34c2f0401b44","entry":"load_model","repo":"mlcommons/tiny","repo_kind":"official","path":"benchmark/training/anomaly_detection/keras_model.py","file_url":"https://github.com/mlcommons/tiny/blob/HEAD/benchmark/training/anomaly_detection/keras_model.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"62ef34c2f0401b44"}},{"code_sha256_prefix":"fb79e3cbc6774c55","entry":"predict","repo":"mlcommons/tiny_results_v0.5","repo_kind":"official","path":"closed/PengChengLaboratory/code/ad/tiny/v0.1/training/keyword_spotting/eval_quantized_model.py","file_url":"https://github.com/mlcommons/tiny_results_v0.5/blob/HEAD/closed/PengChengLaboratory/code/ad/tiny/v0.1/training/keyword_spotting/eval_quantized_model.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"fb79e3cbc6774c55"}},{"code_sha256_prefix":"b35b49d0dc790d3d","entry":"prepare_model_settings","repo":"mlcommons/tiny_results_v0.5","repo_kind":"official","path":"closed/PengChengLaboratory/code/ad/tiny/v0.1/training/keyword_spotting/keras_model.py","file_url":"https://github.com/mlcommons/tiny_results_v0.5/blob/HEAD/closed/PengChengLaboratory/code/ad/tiny/v0.1/training/keyword_spotting/keras_model.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"b35b49d0dc790d3d"}},{"code_sha256_prefix":"80bcdb8ca9416120","entry":"process_dutycycle","repo":"mlcommons/tiny","repo_kind":"official","path":"benchmark/runner/streaming_ww_utils.py","file_url":"https://github.com/mlcommons/tiny/blob/HEAD/benchmark/runner/streaming_ww_utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"80bcdb8ca9416120"}},{"code_sha256_prefix":"53c9956822b2012c","entry":"process_timestamps","repo":"mlcommons/tiny","repo_kind":"official","path":"benchmark/runner/streaming_ww_utils.py","file_url":"https://github.com/mlcommons/tiny/blob/HEAD/benchmark/runner/streaming_ww_utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"53c9956822b2012c"}},{"code_sha256_prefix":"958a1150a034c5e4","entry":"resnet_v1_eembc","repo":"mlcommons/tiny","repo_kind":"official","path":"benchmark/training/image_classification/keras_model.py","file_url":"https://github.com/mlcommons/tiny/blob/HEAD/benchmark/training/image_classification/keras_model.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"958a1150a034c5e4"}},{"code_sha256_prefix":"b5b3bd1958d30306","entry":"send_cmd_get_resp","repo":"mlcommons/tiny","repo_kind":"official","path":"benchmark/runner/stream_wav_uart.py","file_url":"https://github.com/mlcommons/tiny/blob/HEAD/benchmark/runner/stream_wav_uart.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"b5b3bd1958d30306"}},{"code_sha256_prefix":"54aa90f290420a50","entry":"toByteBuffer","repo":"mlcommons/tiny_results_v0.5","repo_kind":"official","path":"closed/PengChengLaboratory/code/ad/ConvertModel.py","file_url":"https://github.com/mlcommons/tiny_results_v0.5/blob/HEAD/closed/PengChengLaboratory/code/ad/ConvertModel.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"54aa90f290420a50"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}