{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/mlfmf-data-sets-for-machine-learning-for-1","title":"MLFMF: Data Sets for Machine Learning for Mathematical Formalization","arxiv_id":"2310.16005","date":"2023-10-24","proceeding":"NeurIPS 2023 11","authors":["Andrej Bauer","Matej Petković","Ljupčo Todorovski"],"abstract":"We introduce MLFMF, a collection of data sets for benchmarking recommendation systems used to support formalization of mathematics with proof assistants. These systems help humans identify which previous entries (theorems, constructions, datatypes, and postulates) are relevant in proving a new theorem or carrying out a new construction. Each data set is derived from a library of formalized mathematics written in proof assistants Agda or Lean. The collection includes the largest Lean~4 library Mathlib, and some of the largest Agda libraries: the standard library, the library of univalent mathematics Agda-unimath, and the TypeTopology library. Each data set represents the corresponding library in two ways: as a heterogeneous network, and as a list of s-expressions representing the syntax trees of all the entries in the library. The network contains the (modular) structure of the library and the references between entries, while the s-expressions give complete and easily parsed information about every entry. We report baseline results using standard graph and word embeddings, tree ensembles, and instance-based learning algorithms. The MLFMF data sets provide solid benchmarking support for further investigation of the numerous machine learning approaches to formalized mathematics. The methodology used to extract the networks and the s-expressions readily applies to other libraries, and is applicable to other proof assistants. With more than $250\\,000$ entries in total, this is currently the largest collection of formalized mathematical knowledge in machine learnable format.","url_abs":"https://arxiv.org/abs/2310.16005v1","url_pdf":"https://arxiv.org/pdf/2310.16005v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"mlfmf-data-sets-for-machine-learning-for-1","repo_url":"https://github.com/ul-fmf/mlfmf-data","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok","spdx":"BSD-2-Clause"}}],"tasks":[{"task_slug":"benchmarking","task_name":"Benchmarking"},{"task_slug":"recommendation-systems","task_name":"Recommendation Systems"},{"task_slug":"word-embeddings","task_name":"Word Embeddings"}],"methods":[{"method_slug":null,"method_name":"Library"},{"method_slug":"set","method_name":"SET"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2310.16005","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2310.16005"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/ul-fmf/mlfmf-data","reach":{"status":"ok","spdx":"BSD-2-Clause"}},{"provenance":"deterministic:regex_extraction","url":"https://github.com/eliorc/node2vec","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":3,"unverified":1},"by_repo_kind":{"official":{"samples":3,"ran":2,"repositories":1},"found_in_text":{"samples":1,"ran":1,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"b62bba421eb2c1ca","entry":"load_entries","repo":"ul-fmf/mlfmf-data","repo_kind":"official","path":"light_weight_data_loader.py","file_url":"https://github.com/ul-fmf/mlfmf-data/blob/HEAD/light_weight_data_loader.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-2-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"b62bba421eb2c1ca"}},{"code_sha256_prefix":"cf6710c9a0d5d184","entry":"load_entry","repo":"ul-fmf/mlfmf-data","repo_kind":"official","path":"light_weight_data_loader.py","file_url":"https://github.com/ul-fmf/mlfmf-data/blob/HEAD/light_weight_data_loader.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-2-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"cf6710c9a0d5d184"}},{"code_sha256_prefix":"2677dd8e78ce9f62","entry":"parallel_generate_walks","repo":"eliorc/node2vec","repo_kind":"found_in_text","path":"node2vec/parallel.py","file_url":"https://github.com/eliorc/node2vec/blob/HEAD/node2vec/parallel.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"2677dd8e78ce9f62"}},{"code_sha256_prefix":"55319c3a809b2006","entry":"load_entry_optimized","repo":"ul-fmf/mlfmf-data","repo_kind":"official","path":"light_weight_data_loader.py","file_url":"https://github.com/ul-fmf/mlfmf-data/blob/HEAD/light_weight_data_loader.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-2-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"55319c3a809b2006"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}