{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/trigo-benchmarking-formal-mathematical-proof","title":"TRIGO: Benchmarking Formal Mathematical Proof Reduction for Generative Language Models","arxiv_id":"2310.10180","date":"2023-10-16","proceeding":null,"authors":["Jing Xiong","Jianhao Shen","Ye Yuan","Haiming Wang","Yichun Yin","Zhengying Liu","Lin Li","Zhijiang Guo","Qingxing Cao","Yinya Huang","Chuanyang Zheng","Xiaodan Liang","Ming Zhang","Qun Liu"],"abstract":"Automated theorem proving (ATP) has become an appealing domain for exploring the reasoning ability of the recent successful generative language models. However, current ATP benchmarks mainly focus on symbolic inference, but rarely involve the understanding of complex number combination reasoning. In this work, we propose TRIGO, an ATP benchmark that not only requires a model to reduce a trigonometric expression with step-by-step proofs but also evaluates a generative LM's reasoning ability on formulas and its capability to manipulate, group, and factor number terms. We gather trigonometric expressions and their reduced forms from the web, annotate the simplification process manually, and translate it into the Lean formal language system. We then automatically generate additional examples from the annotated samples to expand the dataset. Furthermore, we develop an automatic generator based on Lean-Gym to create dataset splits of varying difficulties and distributions in order to thoroughly analyze the model's generalization ability. Our extensive experiments show our proposed TRIGO poses a new challenge for advanced generative LM's including GPT-4 which is pre-trained on a considerable amount of open-source formal theorem-proving language data, and provide a new tool to study the generative LM's ability on both formal and mathematical reasoning.","url_abs":"https://arxiv.org/abs/2310.10180v2","url_pdf":"https://arxiv.org/pdf/2310.10180v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"trigo-benchmarking-formal-mathematical-proof","repo_url":"https://github.com/menik1126/TRIGO","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"none","reach":{"status":"ok"}}],"tasks":[{"task_slug":"automated-theorem-proving","task_name":"Automated Theorem Proving"},{"task_slug":"benchmarking","task_name":"Benchmarking"},{"task_slug":"mathematical-reasoning","task_name":"Mathematical Reasoning"}],"methods":[{"method_slug":"absolute-position-encodings","method_name":"Absolute Position Encodings"},{"method_slug":"adam","method_name":"Adam"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"bpe","method_name":"BPE"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"focus","method_name":"Focus"},{"method_slug":"gpt-4","method_name":"GPT-4"},{"method_slug":"label-smoothing","method_name":"Label Smoothing"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"position-wise-feed-forward-layer","method_name":"Position-Wise Feed-Forward Layer"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"transformer","method_name":"Transformer"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2310.10180","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2310.10180"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/menik1126/TRIGO","reach":{"status":"ok"}}],"summary":{"ran":7,"unverified":4},"by_repo_kind":{"official":{"samples":11,"ran":7,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":11,"samples":[{"code_sha256_prefix":"b1f565f85a0a30f7","entry":"get_dataloader_batced","repo":"menik1126/TRIGO","repo_kind":"official","path":"auto_generate_atp/data_manager.py","file_url":"https://github.com/menik1126/TRIGO/blob/HEAD/auto_generate_atp/data_manager.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"b1f565f85a0a30f7"}},{"code_sha256_prefix":"618ea61c989279b7","entry":"get_test_dataloader","repo":"menik1126/TRIGO","repo_kind":"official","path":"auto_generate_atp/data_manager.py","file_url":"https://github.com/menik1126/TRIGO/blob/HEAD/auto_generate_atp/data_manager.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"618ea61c989279b7"}},{"code_sha256_prefix":"2a5d8406a5fc673f","entry":"mytrig2trig","repo":"menik1126/TRIGO","repo_kind":"official","path":"auto_generate_atp/utils_trigo.py","file_url":"https://github.com/menik1126/TRIGO/blob/HEAD/auto_generate_atp/utils_trigo.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"2a5d8406a5fc673f"}},{"code_sha256_prefix":"702edc3e488e9212","entry":"process_const_replacement","repo":"menik1126/TRIGO","repo_kind":"official","path":"auto_generate_atp/generate_lemma_from_trigo.py","file_url":"https://github.com/menik1126/TRIGO/blob/HEAD/auto_generate_atp/generate_lemma_from_trigo.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"702edc3e488e9212"}},{"code_sha256_prefix":"c61d1fab051dd610","entry":"read_data","repo":"menik1126/TRIGO","repo_kind":"official","path":"auto_generate_atp/delete_useless_data_trigo.py","file_url":"https://github.com/menik1126/TRIGO/blob/HEAD/auto_generate_atp/delete_useless_data_trigo.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"c61d1fab051dd610"}},{"code_sha256_prefix":"610350f2e82667bb","entry":"simplify_pi_parts_without_evaluating_trig","repo":"menik1126/TRIGO","repo_kind":"official","path":"auto_generate_atp/generate_lemma_from_trigo.py","file_url":"https://github.com/menik1126/TRIGO/blob/HEAD/auto_generate_atp/generate_lemma_from_trigo.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"610350f2e82667bb"}},{"code_sha256_prefix":"5b6fdf79afd52535","entry":"trig2mytrig","repo":"menik1126/TRIGO","repo_kind":"official","path":"auto_generate_atp/utils_trigo.py","file_url":"https://github.com/menik1126/TRIGO/blob/HEAD/auto_generate_atp/utils_trigo.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"5b6fdf79afd52535"}},{"code_sha256_prefix":"171ebe0ec0768910","entry":"cal_branches","repo":"menik1126/TRIGO","repo_kind":"official","path":"auto_generate_atp/data_analysis.py","file_url":"https://github.com/menik1126/TRIGO/blob/HEAD/auto_generate_atp/data_analysis.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"171ebe0ec0768910"}},{"code_sha256_prefix":"2b3a7c4812750587","entry":"get_denominators","repo":"menik1126/TRIGO","repo_kind":"official","path":"auto_generate_atp/utils_trigo.py","file_url":"https://github.com/menik1126/TRIGO/blob/HEAD/auto_generate_atp/utils_trigo.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"2b3a7c4812750587"}},{"code_sha256_prefix":"f00580db9b9f415c","entry":"read_data","repo":"menik1126/TRIGO","repo_kind":"official","path":"auto_generate_atp/generate_train_data_trigo_raw.py","file_url":"https://github.com/menik1126/TRIGO/blob/HEAD/auto_generate_atp/generate_train_data_trigo_raw.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"f00580db9b9f415c"}},{"code_sha256_prefix":"dcca5f6301a3b812","entry":"strToInfix","repo":"menik1126/TRIGO","repo_kind":"official","path":"auto_generate_atp/generate_lemma_from_trigo.py","file_url":"https://github.com/menik1126/TRIGO/blob/HEAD/auto_generate_atp/generate_lemma_from_trigo.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"dcca5f6301a3b812"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}