{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/metrex-a-benchmark-for-verilog-code-metric","title":"MetRex: A Benchmark for Verilog Code Metric Reasoning Using LLMs","arxiv_id":"2411.03471","date":"2024-11-05","proceeding":null,"authors":["Manar Abdelatty","Jingxiao Ma","Sherief Reda"],"abstract":"Large Language Models (LLMs) have been applied to various hardware design tasks, including Verilog code generation, EDA tool scripting, and RTL bug fixing. Despite this extensive exploration, LLMs are yet to be used for the task of post-synthesis metric reasoning and estimation of HDL designs. In this paper, we assess the ability of LLMs to reason about post-synthesis metrics of Verilog designs. We introduce MetRex, a large-scale dataset comprising 25,868 Verilog HDL designs and their corresponding post-synthesis metrics, namely area, delay, and static power. MetRex incorporates a Chain of Thought (CoT) template to enhance LLMs' reasoning about these metrics. Extensive experiments show that Supervised Fine-Tuning (SFT) boosts the LLM's reasoning capabilities on average by 37.0\\%, 25.3\\%, and 25.7\\% on the area, delay, and static power, respectively. While SFT improves performance on our benchmark, it remains far from achieving optimal results, especially on complex problems. Comparing to state-of-the-art regression models, our approach delivers accurate post-synthesis predictions for 17.4\\% more designs (within a 5\\% error margin), in addition to offering a 1.7x speedup by eliminating the need for pre-processing. This work lays the groundwork for advancing LLM-based Verilog code metric reasoning.","url_abs":"https://arxiv.org/abs/2411.03471v2","url_pdf":"https://arxiv.org/pdf/2411.03471v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"metrex-a-benchmark-for-verilog-code-metric","repo_url":"https://github.com/scale-lab/MetRex","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":"bug-fixing","task_name":"Bug fixing"},{"task_slug":"code-generation","task_name":"Code Generation"}],"methods":[{"method_slug":"sft","method_name":"SFT"}],"datasets_introduced":[{"slug":"metrex","name":"MetRex","full_name":""}],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2411.03471","atlas_url":"https://app.syntology.ai/?focus=2411.03471","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2411.03471"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/scale-lab/MetRex","reach":{"status":"ok"}}],"summary":{"unverified":13},"by_repo_kind":{"official":{"samples":13,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":13,"samples":[{"code_sha256_prefix":"5f145285b869418a","entry":"combine_cfgs","repo":"scale-lab/MetRex","repo_kind":"official","path":"src/config/config.py","file_url":"https://github.com/scale-lab/MetRex/blob/HEAD/src/config/config.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"5f145285b869418a"}},{"code_sha256_prefix":"937691c80ff7c36a","entry":"compute_mre","repo":"scale-lab/MetRex","repo_kind":"official","path":"src/model/metrics.py","file_url":"https://github.com/scale-lab/MetRex/blob/HEAD/src/model/metrics.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"937691c80ff7c36a"}},{"code_sha256_prefix":"73f05e7deab9b24a","entry":"create_area_cot","repo":"scale-lab/MetRex","repo_kind":"official","path":"src/synth/cot.py","file_url":"https://github.com/scale-lab/MetRex/blob/HEAD/src/synth/cot.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"73f05e7deab9b24a"}},{"code_sha256_prefix":"88b94f732c80ee9e","entry":"create_delay_cot","repo":"scale-lab/MetRex","repo_kind":"official","path":"src/synth/cot.py","file_url":"https://github.com/scale-lab/MetRex/blob/HEAD/src/synth/cot.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"88b94f732c80ee9e"}},{"code_sha256_prefix":"e7ffe6a11b7c5796","entry":"create_synth_thought","repo":"scale-lab/MetRex","repo_kind":"official","path":"src/synth/cot.py","file_url":"https://github.com/scale-lab/MetRex/blob/HEAD/src/synth/cot.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"e7ffe6a11b7c5796"}},{"code_sha256_prefix":"7b34b7bcd774e5d0","entry":"generate","repo":"scale-lab/MetRex","repo_kind":"official","path":"src/model/model.py","file_url":"https://github.com/scale-lab/MetRex/blob/HEAD/src/model/model.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"7b34b7bcd774e5d0"}},{"code_sha256_prefix":"4804099ebf1f246c","entry":"get_logger","repo":"scale-lab/MetRex","repo_kind":"official","path":"src/utils.py","file_url":"https://github.com/scale-lab/MetRex/blob/HEAD/src/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"4804099ebf1f246c"}},{"code_sha256_prefix":"d4e29831be0745a6","entry":"parse_answer_mistral","repo":"scale-lab/MetRex","repo_kind":"official","path":"src/utils.py","file_url":"https://github.com/scale-lab/MetRex/blob/HEAD/src/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"d4e29831be0745a6"}},{"code_sha256_prefix":"71f27e6804c64e84","entry":"parse_area","repo":"scale-lab/MetRex","repo_kind":"official","path":"src/synth/genus_cot.py","file_url":"https://github.com/scale-lab/MetRex/blob/HEAD/src/synth/genus_cot.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"71f27e6804c64e84"}},{"code_sha256_prefix":"69e18eef992616b9","entry":"parse_delay","repo":"scale-lab/MetRex","repo_kind":"official","path":"src/synth/genus_cot.py","file_url":"https://github.com/scale-lab/MetRex/blob/HEAD/src/synth/genus_cot.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"69e18eef992616b9"}},{"code_sha256_prefix":"ad7ea4e07329dd72","entry":"parse_metrics","repo":"scale-lab/MetRex","repo_kind":"official","path":"src/utils.py","file_url":"https://github.com/scale-lab/MetRex/blob/HEAD/src/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"ad7ea4e07329dd72"}},{"code_sha256_prefix":"c50955f73dfb142d","entry":"parse_power","repo":"scale-lab/MetRex","repo_kind":"official","path":"src/synth/genus_cot.py","file_url":"https://github.com/scale-lab/MetRex/blob/HEAD/src/synth/genus_cot.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"c50955f73dfb142d"}},{"code_sha256_prefix":"8543df2964e03d8e","entry":"test_model","repo":"scale-lab/MetRex","repo_kind":"official","path":"src/model/model.py","file_url":"https://github.com/scale-lab/MetRex/blob/HEAD/src/model/model.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"8543df2964e03d8e"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}