{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/natural-language-is-not-enough-benchmarking","title":"Natural language is not enough: Benchmarking multi-modal generative AI for Verilog generation","arxiv_id":"2407.08473","date":"2024-07-11","proceeding":null,"authors":["Kaiyan Chang","Zhirong Chen","Yunhao Zhou","Wenlong Zhu","Kun Wang","Haobo Xu","Cangyuan Li","Mengdi Wang","Shengwen Liang","Huawei Li","Yinhe Han","Ying Wang"],"abstract":"Natural language interfaces have exhibited considerable potential in the automation of Verilog generation derived from high-level specifications through the utilization of large language models, garnering significant attention. Nevertheless, this paper elucidates that visual representations contribute essential contextual information critical to design intent for hardware architectures possessing spatial complexity, potentially surpassing the efficacy of natural-language-only inputs. Expanding upon this premise, our paper introduces an open-source benchmark for multi-modal generative models tailored for Verilog synthesis from visual-linguistic inputs, addressing both singular and complex modules. Additionally, we introduce an open-source visual and natural language Verilog query language framework to facilitate efficient and user-friendly multi-modal queries. To evaluate the performance of the proposed multi-modal hardware generative AI in Verilog generation tasks, we compare it with a popular method that relies solely on natural language. Our results demonstrate a significant accuracy improvement in the multi-modal generated Verilog compared to queries based solely on natural language. We hope to reveal a new approach to hardware design in the large-hardware-design-model era, thereby fostering a more diversified and productive approach to hardware design.","url_abs":"https://arxiv.org/abs/2407.08473v1","url_pdf":"https://arxiv.org/pdf/2407.08473v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"natural-language-is-not-enough-benchmarking","repo_url":"https://github.com/aichipdesign/chipgptv","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"none","reach":{"status":"ok"}}],"tasks":[{"task_slug":"benchmarking","task_name":"Benchmarking"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2407.08473","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2407.08473"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/aichipdesign/chipgptv","reach":{"status":"ok"}}],"summary":{"ran":8,"unverified":1},"by_repo_kind":{"official":{"samples":9,"ran":8,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":9,"samples":[{"code_sha256_prefix":"e024229cecdb8deb","entry":"extract_snippet","repo":"aichipdesign/chipgptv","repo_kind":"official","path":"benchmark_exp/next_token_correctness.py","file_url":"https://github.com/aichipdesign/chipgptv/blob/HEAD/benchmark_exp/next_token_correctness.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"e024229cecdb8deb"}},{"code_sha256_prefix":"7f919b8427f46b43","entry":"find_reference_file","repo":"aichipdesign/chipgptv","repo_kind":"official","path":"benchmark_exp/next_token_correctness.py","file_url":"https://github.com/aichipdesign/chipgptv/blob/HEAD/benchmark_exp/next_token_correctness.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"7f919b8427f46b43"}},{"code_sha256_prefix":"09d4d4a742ac61f7","entry":"generate_code","repo":"aichipdesign/chipgptv","repo_kind":"official","path":"test_benchmark/rtlcoder_finetune_benchmark.py","file_url":"https://github.com/aichipdesign/chipgptv/blob/HEAD/test_benchmark/rtlcoder_finetune_benchmark.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"09d4d4a742ac61f7"}},{"code_sha256_prefix":"86c8fe8a763c73fc","entry":"get_next_token_from_reference","repo":"aichipdesign/chipgptv","repo_kind":"official","path":"benchmark_exp/next_token_correctness.py","file_url":"https://github.com/aichipdesign/chipgptv/blob/HEAD/benchmark_exp/next_token_correctness.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"86c8fe8a763c73fc"}},{"code_sha256_prefix":"06effc4d2820368c","entry":"get_verilog_code","repo":"aichipdesign/chipgptv","repo_kind":"official","path":"benchmark_exp/generate_code_completion.py","file_url":"https://github.com/aichipdesign/chipgptv/blob/HEAD/benchmark_exp/generate_code_completion.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"06effc4d2820368c"}},{"code_sha256_prefix":"d420994518c4db3d","entry":"modify_and_append","repo":"aichipdesign/chipgptv","repo_kind":"official","path":"benchmark_exp/generate_code_completion.py","file_url":"https://github.com/aichipdesign/chipgptv/blob/HEAD/benchmark_exp/generate_code_completion.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"d420994518c4db3d"}},{"code_sha256_prefix":"57c745aff866bb20","entry":"remove_sentence_with_word","repo":"aichipdesign/chipgptv","repo_kind":"official","path":"benchmark_exp/generate_design_description.py","file_url":"https://github.com/aichipdesign/chipgptv/blob/HEAD/benchmark_exp/generate_design_description.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"57c745aff866bb20"}},{"code_sha256_prefix":"c4ee6bdac8a5fc4a","entry":"test_design","repo":"aichipdesign/chipgptv","repo_kind":"official","path":"benchmark_exp/function_correctness.py","file_url":"https://github.com/aichipdesign/chipgptv/blob/HEAD/benchmark_exp/function_correctness.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"c4ee6bdac8a5fc4a"}},{"code_sha256_prefix":"5ec27caa35181009","entry":"load_model","repo":"aichipdesign/chipgptv","repo_kind":"official","path":"test_benchmark/rtlcoder_finetune_benchmark.py","file_url":"https://github.com/aichipdesign/chipgptv/blob/HEAD/test_benchmark/rtlcoder_finetune_benchmark.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"5ec27caa35181009"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}