{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/arxiv-2605-30358","title":"QASM-Eval: A Dataset to Train and Evaluate LLMs on OpenQASM-3 Beyond Quantum Circuits","arxiv_id":"2605.30358","date":"2026-04-28","proceeding":null,"authors":["Zhenxiao Fu","Lei Jiang","Fan Chen"],"abstract":"Quantum computing remains in the Noisy Intermediate-Scale Quantum (NISQ) era, where the performance is highly constrained to noise. Addressing the limitation often requires hardware-facing capabilities beyond gate-sequence circuit specification, including mid-circuit measurement and classical feedback for quantum error correction (QEC), precise timing control for dynamical decoupling (DD), and pulse-level waveform access for calibration. OpenQASM-3 was introduced to expose exactly these capabilities, providing a hardware-level programming interface. However, despite the rapid progress of large language models in code generation, there is still no dataset specifically designed to train and evaluate LLMs on OpenQASM-3 programs that involve its advanced hardware-oriented features. To address this gap, we introduce QASM-Eval, the first comprehensive dataset designed to train and evaluate LLMs on OpenQASM-3. Rather than focusing on quantum algorithm design or reasoning, QASM-Eval explicitly targets the language's hardware-facing features. QASM-Eval comprises an expert-verified test set of 100 tasks and a training set of 4,000 tasks, systematically covering classical logic, timing scheduling, pulse control, and complex real-world workflows. To automatically validate generated programs, we check syntax, quantum states and program timeline using an extended verifier. Our evaluation reveals that while state-of-the-art LLMs struggle heavily in OpenQASM-3 coding tasks, targeted fine-tuning on QASM-Eval yields significant gains. QASM-Eval provides a crucial benchmark and training foundation to accelerate the development of reliable LLM assistants for hardware-facing quantum programming in NISQ era. Data and code: https://github.com/fuzhenxiao/QASM-Eval","url_abs":"https://arxiv.org/abs/2605.30358","url_pdf":"https://arxiv.org/pdf/2605.30358","source":{"archive":null,"snapshot":"2025-07-28","note":"not in the Papers with Code archive (frozen at the snapshot)","row_kind":"graph","title_abstract_authors_date":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)"},"code_links":[],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":null,"mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2605.30358"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"mentioned_in_github":null,"is_official":null,"provenance":"deterministic:regex_extraction","mentioned_in_paper":null,"url":"https://github.com/fuzhenxiao/QASM-Eval","reach":null}],"summary":{"ran":19,"unverified":4},"by_repo_kind":{"found_in_text":{"samples":23,"ran":19,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":23,"samples":[{"code_sha256_prefix":"59e5f96badbeb49a","entry":"build_one_complex_background","repo":"fuzhenxiao/QASM-Eval","repo_kind":"found_in_text","path":"dataset_factory/background_complex.py","file_url":"https://github.com/fuzhenxiao/QASM-Eval/blob/HEAD/dataset_factory/background_complex.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"59e5f96badbeb49a"}},{"code_sha256_prefix":"0c17e2f40a5a8075","entry":"build_one_pulse_background","repo":"fuzhenxiao/QASM-Eval","repo_kind":"found_in_text","path":"dataset_factory/background_pulse.py","file_url":"https://github.com/fuzhenxiao/QASM-Eval/blob/HEAD/dataset_factory/background_pulse.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"0c17e2f40a5a8075"}},{"code_sha256_prefix":"bfd9e4a2c3354d03","entry":"detect_newline","repo":"fuzhenxiao/QASM-Eval","repo_kind":"found_in_text","path":"dataset_factory/generate_prompt_train.py","file_url":"https://github.com/fuzhenxiao/QASM-Eval/blob/HEAD/dataset_factory/generate_prompt_train.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"bfd9e4a2c3354d03"}},{"code_sha256_prefix":"15a125850ac6f7ab","entry":"extract_context_for_model","repo":"fuzhenxiao/QASM-Eval","repo_kind":"found_in_text","path":"dataset_factory/generate_prompt_train.py","file_url":"https://github.com/fuzhenxiao/QASM-Eval/blob/HEAD/dataset_factory/generate_prompt_train.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"15a125850ac6f7ab"}},{"code_sha256_prefix":"e34681c6ec5cc373","entry":"fmt_angle","repo":"fuzhenxiao/QASM-Eval","repo_kind":"found_in_text","path":"dataset_factory/background_classical.py","file_url":"https://github.com/fuzhenxiao/QASM-Eval/blob/HEAD/dataset_factory/background_classical.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"e34681c6ec5cc373"}},{"code_sha256_prefix":"b60d7699ea0bfa77","entry":"load_json","repo":"fuzhenxiao/QASM-Eval","repo_kind":"found_in_text","path":"dataset_factory/build_parquet_dataset.py","file_url":"https://github.com/fuzhenxiao/QASM-Eval/blob/HEAD/dataset_factory/build_parquet_dataset.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"b60d7699ea0bfa77"}},{"code_sha256_prefix":"5b12edc2dc932c40","entry":"load_meta","repo":"fuzhenxiao/QASM-Eval","repo_kind":"found_in_text","path":"dataset_factory/pulse_coretasks.py","file_url":"https://github.com/fuzhenxiao/QASM-Eval/blob/HEAD/dataset_factory/pulse_coretasks.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"5b12edc2dc932c40"}},{"code_sha256_prefix":"504dc1d7fdef4323","entry":"n_qubits","repo":"fuzhenxiao/QASM-Eval","repo_kind":"found_in_text","path":"dataset_factory/classical_coretasks.py","file_url":"https://github.com/fuzhenxiao/QASM-Eval/blob/HEAD/dataset_factory/classical_coretasks.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"504dc1d7fdef4323"}},{"code_sha256_prefix":"ca3c8b54e895d336","entry":"n_qubits","repo":"fuzhenxiao/QASM-Eval","repo_kind":"found_in_text","path":"dataset_factory/complex_coretasks.py","file_url":"https://github.com/fuzhenxiao/QASM-Eval/blob/HEAD/dataset_factory/complex_coretasks.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"ca3c8b54e895d336"}},{"code_sha256_prefix":"9683abdaccc67842","entry":"n_qubits","repo":"fuzhenxiao/QASM-Eval","repo_kind":"found_in_text","path":"dataset_factory/pulse_coretasks.py","file_url":"https://github.com/fuzhenxiao/QASM-Eval/blob/HEAD/dataset_factory/pulse_coretasks.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"9683abdaccc67842"}},{"code_sha256_prefix":"c077843ed2a79386","entry":"pick_qubit","repo":"fuzhenxiao/QASM-Eval","repo_kind":"found_in_text","path":"dataset_factory/classical_coretasks.py","file_url":"https://github.com/fuzhenxiao/QASM-Eval/blob/HEAD/dataset_factory/classical_coretasks.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"c077843ed2a79386"}},{"code_sha256_prefix":"9ee6413fcd29c289","entry":"pick_qubit","repo":"fuzhenxiao/QASM-Eval","repo_kind":"found_in_text","path":"dataset_factory/complex_coretasks.py","file_url":"https://github.com/fuzhenxiao/QASM-Eval/blob/HEAD/dataset_factory/complex_coretasks.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"9ee6413fcd29c289"}},{"code_sha256_prefix":"3ed4e31d44ffb0c0","entry":"pick_two_distinct","repo":"fuzhenxiao/QASM-Eval","repo_kind":"found_in_text","path":"dataset_factory/background_classical.py","file_url":"https://github.com/fuzhenxiao/QASM-Eval/blob/HEAD/dataset_factory/background_classical.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"3ed4e31d44ffb0c0"}},{"code_sha256_prefix":"03967fc4b0700a50","entry":"prefix_dir_with_prompt","repo":"fuzhenxiao/QASM-Eval","repo_kind":"found_in_text","path":"dataset_factory/generate_prompt_train.py","file_url":"https://github.com/fuzhenxiao/QASM-Eval/blob/HEAD/dataset_factory/generate_prompt_train.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"03967fc4b0700a50"}},{"code_sha256_prefix":"09f7389473514b04","entry":"rand_angle","repo":"fuzhenxiao/QASM-Eval","repo_kind":"found_in_text","path":"dataset_factory/background_classical.py","file_url":"https://github.com/fuzhenxiao/QASM-Eval/blob/HEAD/dataset_factory/background_classical.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"09f7389473514b04"}},{"code_sha256_prefix":"1162e7f1352ecaac","entry":"read_text","repo":"fuzhenxiao/QASM-Eval","repo_kind":"found_in_text","path":"dataset_factory/build_parquet_dataset.py","file_url":"https://github.com/fuzhenxiao/QASM-Eval/blob/HEAD/dataset_factory/build_parquet_dataset.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"1162e7f1352ecaac"}},{"code_sha256_prefix":"19e24295596c14d3","entry":"used_qubits","repo":"fuzhenxiao/QASM-Eval","repo_kind":"found_in_text","path":"dataset_factory/classical_coretasks.py","file_url":"https://github.com/fuzhenxiao/QASM-Eval/blob/HEAD/dataset_factory/classical_coretasks.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"19e24295596c14d3"}},{"code_sha256_prefix":"e86dc1a5f823a7d6","entry":"used_qubits","repo":"fuzhenxiao/QASM-Eval","repo_kind":"found_in_text","path":"dataset_factory/complex_coretasks.py","file_url":"https://github.com/fuzhenxiao/QASM-Eval/blob/HEAD/dataset_factory/complex_coretasks.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"e86dc1a5f823a7d6"}},{"code_sha256_prefix":"96c621e0b81d5042","entry":"used_qubits","repo":"fuzhenxiao/QASM-Eval","repo_kind":"found_in_text","path":"dataset_factory/pulse_coretasks.py","file_url":"https://github.com/fuzhenxiao/QASM-Eval/blob/HEAD/dataset_factory/pulse_coretasks.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"96c621e0b81d5042"}},{"code_sha256_prefix":"b35350dd508a219b","entry":"build_dataset","repo":"fuzhenxiao/QASM-Eval","repo_kind":"found_in_text","path":"dataset_factory/generate_dataset.py","file_url":"https://github.com/fuzhenxiao/QASM-Eval/blob/HEAD/dataset_factory/generate_dataset.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"b35350dd508a219b"}},{"code_sha256_prefix":"8662b34ddebd9f07","entry":"generate_prompt_qasm","repo":"fuzhenxiao/QASM-Eval","repo_kind":"found_in_text","path":"dataset_factory/generate_dataset.py","file_url":"https://github.com/fuzhenxiao/QASM-Eval/blob/HEAD/dataset_factory/generate_dataset.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"8662b34ddebd9f07"}},{"code_sha256_prefix":"bcef96eadfa21ac3","entry":"generate_qasm_corpora","repo":"fuzhenxiao/QASM-Eval","repo_kind":"found_in_text","path":"dataset_factory/generate_dataset.py","file_url":"https://github.com/fuzhenxiao/QASM-Eval/blob/HEAD/dataset_factory/generate_dataset.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"bcef96eadfa21ac3"}},{"code_sha256_prefix":"139e3f5cb8d4c521","entry":"strip_task_instruction","repo":"fuzhenxiao/QASM-Eval","repo_kind":"found_in_text","path":"dataset_factory/build_parquet_dataset.py","file_url":"https://github.com/fuzhenxiao/QASM-Eval/blob/HEAD/dataset_factory/build_parquet_dataset.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"139e3f5cb8d4c521"}}]},"arxiv_metadata":{"licence":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)","fields":["title","abstract","authors","date"],"primary_category":"cs.LG","source":"arxiv_2026.jsonl"},"syntology_extracted_results":{"kind":"leaderboard_placements","source":"Syntology's leaderboard-shaped extractor over the paper's own arXiv-HTML tables: a model pointed at a cell, the number was read from that cell and checked against the board's metric, dataset, split and scale, and an independent check accepted the entry; not reviewed by the paper's authors or the archive's editors","extractor_model":"global.anthropic.claude-sonnet-4-5-20250929-v1:0","verifier_model":null,"prompt_sha":"fa63d4bb9d755694","coverage":{"sentence":"Syntology has checked 6,264 of the 9,581 papers on this site that are newer than the archive; results from the others appear after they are checked.","papers_newer_than_archive":9581,"papers_checked":6264},"entries":[],"not_placed":{"boards":0,"rejected_by_independent_check":0,"refused_by_a_rule":0,"check_did_not_answer":0,"proposed_without_a_cell":0,"declined_by_site":0}}}