{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/2305-14516","title":"Chakra: Advancing Performance Benchmarking and Co-design using Standardized Execution Traces","arxiv_id":"2305.14516","date":"2023-05-23","proceeding":null,"authors":["Srinivas Sridharan","Taekyung Heo","Louis Feng","Zhaodong Wang","Matt Bergeron","Wenyin Fu","Shengbao Zheng","Brian Coutinho","Saeed Rashidi","Changhai Man","Tushar Krishna"],"abstract":"Benchmarking and co-design are essential for driving optimizations and innovation around ML models, ML software, and next-generation hardware. Full workload benchmarks, e.g. MLPerf, play an essential role in enabling fair comparison across different software and hardware stacks especially once systems are fully designed and deployed. However, the pace of AI innovation demands a more agile methodology to benchmark creation and usage by simulators and emulators for future system co-design. We propose Chakra, an open graph schema for standardizing workload specification capturing key operations and dependencies, also known as Execution Trace (ET). In addition, we propose a complementary set of tools/capabilities to enable collection, generation, and adoption of Chakra ETs by a wide range of simulators, emulators, and benchmarks. For instance, we use generative AI models to learn latent statistical properties across thousands of Chakra ETs and use these models to synthesize Chakra ETs. These synthetic ETs can obfuscate key proprietary information and also target future what-if scenarios. As an example, we demonstrate an end-to-end proof-of-concept that converts PyTorch ETs to Chakra ETs and uses this to drive an open-source training system simulator (ASTRA-sim). Our end-goal is to build a vibrant industry-wide ecosystem of agile benchmarks and tools to drive future AI system co-design.","url_abs":"https://arxiv.org/abs/2305.14516v2","url_pdf":"https://arxiv.org/pdf/2305.14516v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"2305-14516","repo_url":"https://github.com/chakra-et/chakra","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok","spdx":"Apache-2.0"}},{"paper_slug":"2305-14516","repo_url":"https://github.com/mlcommons/chakra","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":null}],"tasks":[{"task_slug":"benchmarking","task_name":"Benchmarking"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2305.14516","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2305.14516"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/chakra-et/chakra","reach":{"status":"ok","spdx":"Apache-2.0"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/mlcommons/chakra","reach":null}],"summary":{"ran_draft_wrong":1,"unverified":3},"by_repo_kind":{"official":{"samples":3,"ran":0,"repositories":1},"listed":{"samples":1,"ran":1,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"7d7e35dc98ed2ac4","entry":"escape_label","repo":"mlcommons/chakra","repo_kind":"listed","path":"src/visualizer/visualizer.py","file_url":"https://github.com/mlcommons/chakra/blob/HEAD/src/visualizer/visualizer.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"7d7e35dc98ed2ac4"}},{"code_sha256_prefix":"215e046594661350","entry":"get_logger","repo":"chakra-et/chakra","repo_kind":"official","path":"et_converter/et_converter.py","file_url":"https://github.com/chakra-et/chakra/blob/HEAD/et_converter/et_converter.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"215e046594661350"}},{"code_sha256_prefix":"9eefcc5fa427824b","entry":"is_local_mem_node","repo":"chakra-et/chakra","repo_kind":"official","path":"timeline_visualizer/timeline_visualizer.py","file_url":"https://github.com/chakra-et/chakra/blob/HEAD/timeline_visualizer/timeline_visualizer.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"9eefcc5fa427824b"}},{"code_sha256_prefix":"6f9323d40cf7eb39","entry":"is_remote_mem_node","repo":"chakra-et/chakra","repo_kind":"official","path":"timeline_visualizer/timeline_visualizer.py","file_url":"https://github.com/chakra-et/chakra/blob/HEAD/timeline_visualizer/timeline_visualizer.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"6f9323d40cf7eb39"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}