{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/spelling-error-correction-with-soft-masked","title":"Spelling Error Correction with Soft-Masked BERT","arxiv_id":"2005.07421","date":"2020-05-15","proceeding":"ACL 2020 6","authors":["Shaohua Zhang","Haoran Huang","Jicong Liu","Hang Li"],"abstract":"Spelling error correction is an important yet challenging task because a satisfactory solution of it essentially needs human-level language understanding ability. Without loss of generality we consider Chinese spelling error correction (CSC) in this paper. A state-of-the-art method for the task selects a character from a list of candidates for correction (including non-correction) at each position of the sentence on the basis of BERT, the language representation model. The accuracy of the method can be sub-optimal, however, because BERT does not have sufficient capability to detect whether there is an error at each position, apparently due to the way of pre-training it using mask language modeling. In this work, we propose a novel neural architecture to address the aforementioned issue, which consists of a network for error detection and a network for error correction based on BERT, with the former being connected to the latter with what we call soft-masking technique. Our method of using `Soft-Masked BERT' is general, and it may be employed in other language detection-correction problems. Experimental results on two datasets demonstrate that the performance of our proposed method is significantly better than the baselines including the one solely based on BERT.","url_abs":"https://arxiv.org/abs/2005.07421v1","url_pdf":"https://arxiv.org/pdf/2005.07421v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"spelling-error-correction-with-soft-masked","repo_url":"https://github.com/gitabtion/BertBasedCorrectionModels","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}},{"paper_slug":"spelling-error-correction-with-soft-masked","repo_url":"https://github.com/hiyoung123/SoftMaskedBert","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"spelling-error-correction-with-soft-masked","repo_url":"https://github.com/lemon234071/TransformerBaselines","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"spelling-error-correction-with-soft-masked","repo_url":"https://github.com/nocoolsandwich/pinyin_gen_error_sentence","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"spelling-error-correction-with-soft-masked","repo_url":"https://github.com/mszpc/SoftMaskedBert","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"mindspore","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"chinese-spelling-error-correction","task_name":"Chinese Spelling Error Correction"},{"task_slug":"language-modeling","task_name":"Language Modeling"},{"task_slug":"language-modelling","task_name":"Language Modelling"},{"task_slug":null,"task_name":"Position"},{"task_slug":"sentence","task_name":"Sentence"}],"methods":[{"method_slug":"adam","method_name":"Adam"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"attention-dropout","method_name":"Attention Dropout"},{"method_slug":"bert","method_name":"BERT"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"linear-warmup-with-linear-decay","method_name":"Linear Warmup With Linear Decay"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"weight-decay","method_name":"Weight Decay"},{"method_slug":"wordpiece","method_name":"WordPiece"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2005.07421","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2005.07421"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/huggingface/transformers","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/mszpc/SoftMaskedBert","reach":{"status":"ok","spdx":"Apache-2.0"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/hiyoung123/SoftMaskedBert","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/gitabtion/BertBasedCorrectionModels","reach":{"status":"ok","spdx":"Apache-2.0"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/lemon234071/TransformerBaselines","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/nocoolsandwich/pinyin_gen_error_sentence","reach":null}],"summary":{"ran":4,"ran_draft_wrong":1,"unverified":12},"by_repo_kind":{"listed":{"samples":17,"ran":5,"repositories":5}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":3,"samples":[{"code_sha256_prefix":"a728cf7fd8651ddf","entry":"BiGRU","repo":"nocoolsandwich/pinyin_gen_error_sentence","repo_kind":"listed","path":"model/soft_masked_bert.py","file_url":"https://github.com/nocoolsandwich/pinyin_gen_error_sentence/blob/HEAD/model/soft_masked_bert.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"a728cf7fd8651ddf"}},{"code_sha256_prefix":"3e404afd004bb929","entry":"BiGRU","repo":"lemon234071/TransformerBaselines","repo_kind":"listed","path":"agents/bert/soft_masked_bert/soft_masked_bert.py","file_url":"https://github.com/lemon234071/TransformerBaselines/blob/HEAD/agents/bert/soft_masked_bert/soft_masked_bert.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"3e404afd004bb929"}},{"code_sha256_prefix":"b722732fc955e68c","entry":"SoftMaskedBert","repo":"nocoolsandwich/pinyin_gen_error_sentence","repo_kind":"listed","path":"model/soft_masked_bert.py","file_url":"https://github.com/nocoolsandwich/pinyin_gen_error_sentence/blob/HEAD/model/soft_masked_bert.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"b722732fc955e68c"}},{"code_sha256_prefix":"3f71761e83e0177e","entry":"SoftMaskedBert","repo":"hiyoung123/SoftMaskedBert","repo_kind":"listed","path":"model/soft_masked_bert.py","file_url":"https://github.com/hiyoung123/SoftMaskedBert/blob/HEAD/model/soft_masked_bert.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"3f71761e83e0177e"}},{"code_sha256_prefix":"3c707eafb83ce730","entry":"make_optimizer","repo":"gitabtion/BertBasedCorrectionModels","repo_kind":"listed","path":"bbcm/solver/build.py","file_url":"https://github.com/gitabtion/BertBasedCorrectionModels/blob/HEAD/bbcm/solver/build.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"3c707eafb83ce730"}},{"code_sha256_prefix":"0d5cdc01abffb5cb","entry":"SoftMaskedBert","repo":"lemon234071/TransformerBaselines","repo_kind":"listed","path":"agents/bert/soft_masked_bert/soft_masked_bert.py","file_url":"https://github.com/lemon234071/TransformerBaselines/blob/HEAD/agents/bert/soft_masked_bert/soft_masked_bert.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"0d5cdc01abffb5cb"}},{"code_sha256_prefix":"c06aca58c0b60862","entry":"compute_correction_prf","repo":"mszpc/SoftMaskedBert","repo_kind":"listed","path":"softmaskedbert/src/utils.py","file_url":"https://github.com/mszpc/SoftMaskedBert/blob/HEAD/softmaskedbert/src/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"c06aca58c0b60862"}},{"code_sha256_prefix":"98cd0539309793ca","entry":"compute_corrector_prf","repo":"gitabtion/BertBasedCorrectionModels","repo_kind":"listed","path":"bbcm/utils/evaluations.py","file_url":"https://github.com/gitabtion/BertBasedCorrectionModels/blob/HEAD/bbcm/utils/evaluations.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"98cd0539309793ca"}},{"code_sha256_prefix":"8b028660c9e94b5c","entry":"compute_corrector_prf","repo":"mszpc/SoftMaskedBert","repo_kind":"listed","path":"softmaskedbert/src/utils.py","file_url":"https://github.com/mszpc/SoftMaskedBert/blob/HEAD/softmaskedbert/src/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"8b028660c9e94b5c"}},{"code_sha256_prefix":"ac8790d05ecb703c","entry":"compute_detection_prf","repo":"mszpc/SoftMaskedBert","repo_kind":"listed","path":"softmaskedbert/src/utils.py","file_url":"https://github.com/mszpc/SoftMaskedBert/blob/HEAD/softmaskedbert/src/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"ac8790d05ecb703c"}},{"code_sha256_prefix":"b088e76916ad21f3","entry":"compute_sentence_level_prf","repo":"gitabtion/BertBasedCorrectionModels","repo_kind":"listed","path":"bbcm/utils/evaluations.py","file_url":"https://github.com/gitabtion/BertBasedCorrectionModels/blob/HEAD/bbcm/utils/evaluations.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"b088e76916ad21f3"}},{"code_sha256_prefix":"712509715b4485d4","entry":"dump_json","repo":"gitabtion/BertBasedCorrectionModels","repo_kind":"listed","path":"bbcm/utils/file_io.py","file_url":"https://github.com/gitabtion/BertBasedCorrectionModels/blob/HEAD/bbcm/utils/file_io.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"712509715b4485d4"}},{"code_sha256_prefix":"40e10fdbe38a163f","entry":"dump_json","repo":"mszpc/SoftMaskedBert","repo_kind":"listed","path":"softmaskedbert/preprocess_dataset.py","file_url":"https://github.com/mszpc/SoftMaskedBert/blob/HEAD/softmaskedbert/preprocess_dataset.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"40e10fdbe38a163f"}},{"code_sha256_prefix":"9f0e70bc9b57f335","entry":"get_cache_dir","repo":"gitabtion/BertBasedCorrectionModels","repo_kind":"listed","path":"bbcm/utils/file_io.py","file_url":"https://github.com/gitabtion/BertBasedCorrectionModels/blob/HEAD/bbcm/utils/file_io.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"9f0e70bc9b57f335"}},{"code_sha256_prefix":"fa2946b9355af8b5","entry":"load_json","repo":"gitabtion/BertBasedCorrectionModels","repo_kind":"listed","path":"bbcm/utils/file_io.py","file_url":"https://github.com/gitabtion/BertBasedCorrectionModels/blob/HEAD/bbcm/utils/file_io.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"fa2946b9355af8b5"}},{"code_sha256_prefix":"c3ba3217b15fd141","entry":"proc_test_set","repo":"mszpc/SoftMaskedBert","repo_kind":"listed","path":"softmaskedbert/preprocess_dataset.py","file_url":"https://github.com/mszpc/SoftMaskedBert/blob/HEAD/softmaskedbert/preprocess_dataset.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"c3ba3217b15fd141"}},{"code_sha256_prefix":"704deca2af268bca","entry":"report_prf","repo":"gitabtion/BertBasedCorrectionModels","repo_kind":"listed","path":"bbcm/utils/evaluations.py","file_url":"https://github.com/gitabtion/BertBasedCorrectionModels/blob/HEAD/bbcm/utils/evaluations.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"704deca2af268bca"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}