{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/end-to-end-symbolic-regression-with","title":"End-to-end symbolic regression with transformers","arxiv_id":"2204.10532","date":"2022-04-22","proceeding":null,"authors":["Pierre-Alexandre Kamienny","Stéphane d'Ascoli","Guillaume Lample","François Charton"],"abstract":"Symbolic regression, the task of predicting the mathematical expression of a function from the observation of its values, is a difficult task which usually involves a two-step procedure: predicting the \"skeleton\" of the expression up to the choice of numerical constants, then fitting the constants by optimizing a non-convex loss function. The dominant approach is genetic programming, which evolves candidates by iterating this subroutine a large number of times. Neural networks have recently been tasked to predict the correct skeleton in a single try, but remain much less powerful. In this paper, we challenge this two-step procedure, and task a Transformer to directly predict the full mathematical expression, constants included. One can subsequently refine the predicted constants by feeding them to the non-convex optimizer as an informed initialization. We present ablations to show that this end-to-end approach yields better results, sometimes even without the refinement step. We evaluate our model on problems from the SRBench benchmark and show that our model approaches the performance of state-of-the-art genetic programming with several orders of magnitude faster inference.","url_abs":"https://arxiv.org/abs/2204.10532v1","url_pdf":"https://arxiv.org/pdf/2204.10532v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"end-to-end-symbolic-regression-with","repo_url":"https://github.com/facebookresearch/symbolicregression","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"end-to-end-symbolic-regression-with","repo_url":"https://github.com/deep-symbolic-mathematics/TPSR","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"end-to-end-symbolic-regression-with","repo_url":"https://github.com/deep-symbolic-mathematics/Multimodal-Math-Pretraining","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"symbolic-regression","task_name":"Symbolic Regression"},{"task_slug":"regression-1","task_name":"regression"}],"methods":[{"method_slug":"absolute-position-encodings","method_name":"Absolute Position Encodings"},{"method_slug":"adam","method_name":"Adam"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"bpe","method_name":"BPE"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"label-smoothing","method_name":"Label Smoothing"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"position-wise-feed-forward-layer","method_name":"Position-Wise Feed-Forward Layer"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"transformer","method_name":"Transformer"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2204.10532","atlas_url":"https://app.syntology.ai/?focus=2204.10532","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2204.10532"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/facebookresearch/symbolicregression","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/deep-symbolic-mathematics/TPSR","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/deep-symbolic-mathematics/Multimodal-Math-Pretraining","reach":null}],"summary":{"ran_fixture":3,"ran_draft_wrong":1},"by_repo_kind":{"official":{"samples":1,"ran":1,"repositories":1},"listed":{"samples":3,"ran":3,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"a3e27bd1e8b628ed","entry":"Embedding","repo":"deep-symbolic-mathematics/TPSR","repo_kind":"listed","path":"symbolicregression/model/transformer.py","file_url":"https://github.com/deep-symbolic-mathematics/TPSR/blob/HEAD/symbolicregression/model/transformer.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a3e27bd1e8b628ed"}},{"code_sha256_prefix":"b2470f4f0c09db41","entry":"get_masks","repo":"deep-symbolic-mathematics/TPSR","repo_kind":"listed","path":"symbolicregression/model/transformer.py","file_url":"https://github.com/deep-symbolic-mathematics/TPSR/blob/HEAD/symbolicregression/model/transformer.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b2470f4f0c09db41"}},{"code_sha256_prefix":"40b1a20fe12de364","entry":"read_file","repo":"facebookresearch/symbolicregression","repo_kind":"official","path":"evaluate.py","file_url":"https://github.com/facebookresearch/symbolicregression/blob/HEAD/evaluate.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"40b1a20fe12de364"}},{"code_sha256_prefix":"f434565e2011ab22","entry":"top_k_top_p_filtering","repo":"deep-symbolic-mathematics/TPSR","repo_kind":"listed","path":"symbolicregression/model/transformer.py","file_url":"https://github.com/deep-symbolic-mathematics/TPSR/blob/HEAD/symbolicregression/model/transformer.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f434565e2011ab22"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}