{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/mathbert-a-pre-trained-language-model-for","title":"MathBERT: A Pre-trained Language Model for General NLP Tasks in Mathematics Education","arxiv_id":"2106.07340","date":"2021-06-02","proceeding":null,"authors":["Jia Tracy Shen","Michiharu Yamashita","Ethan Prihar","Neil Heffernan","Xintao Wu","Ben Graff","Dongwon Lee"],"abstract":"Since the introduction of the original BERT (i.e., BASE BERT), researchers have developed various customized BERT models with improved performance for specific domains and tasks by exploiting the benefits of transfer learning. Due to the nature of mathematical texts, which often use domain specific vocabulary along with equations and math symbols, we posit that the development of a new BERT model for mathematics would be useful for many mathematical downstream tasks. In this resource paper, we introduce our multi-institutional effort (i.e., two learning platforms and three academic institutions in the US) toward this need: MathBERT, a model created by pre-training the BASE BERT model on a large mathematical corpus ranging from pre-kindergarten (pre-k), to high-school, to college graduate level mathematical content. In addition, we select three general NLP tasks that are often used in mathematics education: prediction of knowledge component, auto-grading open-ended Q&A, and knowledge tracing, to demonstrate the superiority of MathBERT over BASE BERT. Our experiments show that MathBERT outperforms prior best methods by 1.2-22% and BASE BERT by 2-8% on these tasks. In addition, we build a mathematics specific vocabulary 'mathVocab' to train with MathBERT. We discover that MathBERT pre-trained with 'mathVocab' outperforms MathBERT trained with the BASE BERT vocabulary (i.e., 'origVocab'). MathBERT is currently being adopted at the participated leaning platforms: Stride, Inc, a commercial educational resource provider, and ASSISTments.org, a free online educational platform. We release MathBERT for public usage at: https://github.com/tbs17/MathBERT.","url_abs":"https://arxiv.org/abs/2106.07340v5","url_pdf":"https://arxiv.org/pdf/2106.07340v5.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"mathbert-a-pre-trained-language-model-for","repo_url":"https://github.com/tbs17/MathBERT","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"tf","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"knowledge-tracing","task_name":"Knowledge Tracing"},{"task_slug":"language-modeling","task_name":"Language Modeling"},{"task_slug":"language-modelling","task_name":"Language Modelling"},{"task_slug":"math","task_name":"Math"},{"task_slug":"transfer-learning","task_name":"Transfer Learning"}],"methods":[{"method_slug":"adam","method_name":"Adam"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"attention-dropout","method_name":"Attention Dropout"},{"method_slug":"bert","method_name":"BERT"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"linear-warmup-with-linear-decay","method_name":"Linear Warmup With Linear Decay"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"weight-decay","method_name":"Weight Decay"},{"method_slug":"wordpiece","method_name":"WordPiece"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2106.07340","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2106.07340"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/tbs17/MathBERT","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran_draft_wrong":2,"ran_honours":1,"unverified":6},"by_repo_kind":{"official":{"samples":9,"ran":3,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":9,"samples":[{"code_sha256_prefix":"1923fc05163d207d","entry":"convert_to_unicode","repo":"tbs17/MathBERT","repo_kind":"official","path":"mathbert/tokenization.py","file_url":"https://github.com/tbs17/MathBERT/blob/HEAD/mathbert/tokenization.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"MIT","inline_ok":false,"mcp_get_code":{"code_sha256":"1923fc05163d207d"}},{"code_sha256_prefix":"d08f324f950148de","entry":"gelu","repo":"tbs17/MathBERT","repo_kind":"official","path":"mathbert/modeling.py","file_url":"https://github.com/tbs17/MathBERT/blob/HEAD/mathbert/modeling.py","link_basis":"harvester_set","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"MIT","inline_ok":false,"mcp_get_code":{"code_sha256":"d08f324f950148de"}},{"code_sha256_prefix":"5842ee67106321d6","entry":"get_activation","repo":"tbs17/MathBERT","repo_kind":"official","path":"mathbert/modeling.py","file_url":"https://github.com/tbs17/MathBERT/blob/HEAD/mathbert/modeling.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":false,"mcp_get_code":{"code_sha256":"5842ee67106321d6"}},{"code_sha256_prefix":"b24b3b9c700da839","entry":"create_float_feature","repo":"tbs17/MathBERT","repo_kind":"official","path":"mathbert/create_pretraining_data.py","file_url":"https://github.com/tbs17/MathBERT/blob/HEAD/mathbert/create_pretraining_data.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":false,"mcp_get_code":{"code_sha256":"b24b3b9c700da839"}},{"code_sha256_prefix":"dd3ef48a15aec0cd","entry":"create_int_feature","repo":"tbs17/MathBERT","repo_kind":"official","path":"mathbert/create_pretraining_data.py","file_url":"https://github.com/tbs17/MathBERT/blob/HEAD/mathbert/create_pretraining_data.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":false,"mcp_get_code":{"code_sha256":"dd3ef48a15aec0cd"}},{"code_sha256_prefix":"1b46c9ce24fc6db3","entry":"file_based_input_fn_builder","repo":"tbs17/MathBERT","repo_kind":"official","path":"mathbert/run_classifier.py","file_url":"https://github.com/tbs17/MathBERT/blob/HEAD/mathbert/run_classifier.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":false,"mcp_get_code":{"code_sha256":"1b46c9ce24fc6db3"}},{"code_sha256_prefix":"50958618b65e514e","entry":"get_assignment_map_from_checkpoint","repo":"tbs17/MathBERT","repo_kind":"official","path":"mathbert/modeling.py","file_url":"https://github.com/tbs17/MathBERT/blob/HEAD/mathbert/modeling.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":false,"mcp_get_code":{"code_sha256":"50958618b65e514e"}},{"code_sha256_prefix":"ff83ccc8b0b6462d","entry":"load_vocab","repo":"tbs17/MathBERT","repo_kind":"official","path":"mathbert/tokenization.py","file_url":"https://github.com/tbs17/MathBERT/blob/HEAD/mathbert/tokenization.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":false,"mcp_get_code":{"code_sha256":"ff83ccc8b0b6462d"}},{"code_sha256_prefix":"0e5615f8994003cf","entry":"printable_text","repo":"tbs17/MathBERT","repo_kind":"official","path":"mathbert/tokenization.py","file_url":"https://github.com/tbs17/MathBERT/blob/HEAD/mathbert/tokenization.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":false,"mcp_get_code":{"code_sha256":"0e5615f8994003cf"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}