{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/holstep-a-machine-learning-dataset-for-higher","title":"HolStep: A Machine Learning Dataset for Higher-order Logic Theorem Proving","arxiv_id":"1703.00426","date":"2017-03-01","proceeding":null,"authors":["Cezary Kaliszyk","François Chollet","Christian Szegedy"],"abstract":"Large computer-understandable proofs consist of millions of intermediate\nlogical steps. The vast majority of such steps originate from manually selected\nand manually guided heuristics applied to intermediate goals. So far, machine\nlearning has generally not been used to filter or generate these steps. In this\npaper, we introduce a new dataset based on Higher-Order Logic (HOL) proofs, for\nthe purpose of developing new machine learning-based theorem-proving\nstrategies. We make this dataset publicly available under the BSD license. We\npropose various machine learning tasks that can be performed on this dataset,\nand discuss their significance for theorem proving. We also benchmark a set of\nsimple baseline machine learning models suited for the tasks (including\nlogistic regression, convolutional neural networks and recurrent neural\nnetworks). The results of our baseline models show the promise of applying\nmachine learning to HOL theorem proving.","url_abs":"http://arxiv.org/abs/1703.00426v1","url_pdf":"http://arxiv.org/pdf/1703.00426v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"holstep-a-machine-learning-dataset-for-higher","repo_url":"https://github.com/tensorflow/deepmath/tree/master/deepmath/holstep_baselines","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"tf","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"automated-theorem-proving","task_name":"Automated Theorem Proving"},{"task_slug":"machine-learning","task_name":"BIG-bench Machine Learning"}],"methods":[],"datasets_introduced":[{"slug":"holstep","name":"HolStep","full_name":""}],"methods_introduced":[],"results":[{"leaderboard":"/sota/automated-theorem-proving-on-holstep","task":"Automated Theorem Proving","dataset":"HolStep (Conditional)","model":"Siamese 1D CNN-LSTM","rank_in_archive_order":4,"of":5,"metrics":{"Classification Accuracy":"0.83"},"uses_additional_data":false},{"leaderboard":"/sota/automated-theorem-proving-on-holstep","task":"Automated Theorem Proving","dataset":"HolStep (Conditional)","model":"Siamese 1D CNN","rank_in_archive_order":5,"of":5,"metrics":{"Classification Accuracy":"0.82"},"uses_additional_data":false},{"leaderboard":"/sota/automated-theorem-proving-on-holstep-1","task":"Automated Theorem Proving","dataset":"HolStep (Unconditional)","model":"1D CNN","rank_in_archive_order":3,"of":4,"metrics":{"Classification Accuracy":"0.83"},"uses_additional_data":false},{"leaderboard":"/sota/automated-theorem-proving-on-holstep-1","task":"Automated Theorem Proving","dataset":"HolStep (Unconditional)","model":"1D CNN-LSTM","rank_in_archive_order":4,"of":4,"metrics":{"Classification Accuracy":"0.83"},"uses_additional_data":false}],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=1703.00426","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"1703.00426"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/tensorflow/deepmath/tree/master/deepmath/holstep_baselines","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"unverified":6},"by_repo_kind":{"listed":{"samples":6,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"2386f65572e7fe8e","entry":"cnn_2x","repo":"tensorflow/deepmath","repo_kind":"listed","path":"deepmath/holstep_baselines/unconditioned_classification_models.py","file_url":"https://github.com/tensorflow/deepmath/blob/HEAD/deepmath/holstep_baselines/unconditioned_classification_models.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"2386f65572e7fe8e"}},{"code_sha256_prefix":"a06b62157ffe97ae","entry":"cnn_2x_lstm","repo":"tensorflow/deepmath","repo_kind":"listed","path":"deepmath/holstep_baselines/unconditioned_classification_models.py","file_url":"https://github.com/tensorflow/deepmath/blob/HEAD/deepmath/holstep_baselines/unconditioned_classification_models.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"a06b62157ffe97ae"}},{"code_sha256_prefix":"f461198e7b5eff25","entry":"cnn_2x_lstm_siamese","repo":"tensorflow/deepmath","repo_kind":"listed","path":"deepmath/holstep_baselines/conditioned_classification_models.py","file_url":"https://github.com/tensorflow/deepmath/blob/HEAD/deepmath/holstep_baselines/conditioned_classification_models.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"f461198e7b5eff25"}},{"code_sha256_prefix":"f0fb09a4d47ad045","entry":"cnn_2x_siamese","repo":"tensorflow/deepmath","repo_kind":"listed","path":"deepmath/holstep_baselines/conditioned_classification_models.py","file_url":"https://github.com/tensorflow/deepmath/blob/HEAD/deepmath/holstep_baselines/conditioned_classification_models.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"f0fb09a4d47ad045"}},{"code_sha256_prefix":"54b557e33761678c","entry":"embedding_logreg","repo":"tensorflow/deepmath","repo_kind":"listed","path":"deepmath/holstep_baselines/unconditioned_classification_models.py","file_url":"https://github.com/tensorflow/deepmath/blob/HEAD/deepmath/holstep_baselines/unconditioned_classification_models.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"54b557e33761678c"}},{"code_sha256_prefix":"a20b96010f184491","entry":"embedding_logreg_siamese","repo":"tensorflow/deepmath","repo_kind":"listed","path":"deepmath/holstep_baselines/conditioned_classification_models.py","file_url":"https://github.com/tensorflow/deepmath/blob/HEAD/deepmath/holstep_baselines/conditioned_classification_models.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"a20b96010f184491"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}