{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/revisiting-grammatical-error-correction","title":"Revisiting Grammatical Error Correction Evaluation and Beyond","arxiv_id":"2211.01635","date":"2022-11-03","proceeding":null,"authors":["Peiyuan Gong","Xuebo Liu","Heyan Huang","Min Zhang"],"abstract":"Pretraining-based (PT-based) automatic evaluation metrics (e.g., BERTScore and BARTScore) have been widely used in several sentence generation tasks (e.g., machine translation and text summarization) due to their better correlation with human judgments over traditional overlap-based methods. Although PT-based methods have become the de facto standard for training grammatical error correction (GEC) systems, GEC evaluation still does not benefit from pretrained knowledge. This paper takes the first step towards understanding and improving GEC evaluation with pretraining. We first find that arbitrarily applying PT-based metrics to GEC evaluation brings unsatisfactory correlation results because of the excessive attention to inessential systems outputs (e.g., unchanged parts). To alleviate the limitation, we propose a novel GEC evaluation metric to achieve the best of both worlds, namely PT-M2 which only uses PT-based metrics to score those corrected parts. Experimental results on the CoNLL14 evaluation task show that PT-M2 significantly outperforms existing methods, achieving a new state-of-the-art result of 0.949 Pearson correlation. Further analysis reveals that PT-M2 is robust to evaluate competitive GEC systems. Source code and scripts are freely available at https://github.com/pygongnlp/PT-M2.","url_abs":"https://arxiv.org/abs/2211.01635v1","url_pdf":"https://arxiv.org/pdf/2211.01635v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"revisiting-grammatical-error-correction","repo_url":"https://github.com/pygongnlp/pt-m2","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"grammatical-error-correction","task_name":"Grammatical Error Correction"},{"task_slug":"machine-translation","task_name":"Machine Translation"},{"task_slug":"sentence","task_name":"Sentence"},{"task_slug":"text-summarization","task_name":"Text Summarization"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2211.01635","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2211.01635"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/pygongnlp/PT-M2","reach":null}],"summary":{"ran":16,"ran_honours":1,"ran_draft_wrong":4,"unverified":10},"by_repo_kind":{"official":{"samples":31,"ran":21,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":31,"samples":[{"code_sha256_prefix":"6685898ec6c99ed8","entry":"best_edit_seq_bf","repo":"pygongnlp/PT-M2","repo_kind":"official","path":"metrics.py","file_url":"https://github.com/pygongnlp/PT-M2/blob/HEAD/metrics.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"6685898ec6c99ed8"}},{"code_sha256_prefix":"8812d1ff4a2d86e2","entry":"comp_f1","repo":"pygongnlp/PT-M2","repo_kind":"official","path":"metrics.py","file_url":"https://github.com/pygongnlp/PT-M2/blob/HEAD/metrics.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"8812d1ff4a2d86e2"}},{"code_sha256_prefix":"a04366d2f95bd02b","entry":"comp_p","repo":"pygongnlp/PT-M2","repo_kind":"official","path":"metrics.py","file_url":"https://github.com/pygongnlp/PT-M2/blob/HEAD/metrics.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"a04366d2f95bd02b"}},{"code_sha256_prefix":"9602afb1b61091f7","entry":"comp_r","repo":"pygongnlp/PT-M2","repo_kind":"official","path":"metrics.py","file_url":"https://github.com/pygongnlp/PT-M2/blob/HEAD/metrics.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"9602afb1b61091f7"}},{"code_sha256_prefix":"6e7fcf15760cef34","entry":"edit_graph","repo":"pygongnlp/PT-M2","repo_kind":"official","path":"metrics.py","file_url":"https://github.com/pygongnlp/PT-M2/blob/HEAD/metrics.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"6e7fcf15760cef34"}},{"code_sha256_prefix":"98662233319d385a","entry":"equals_ignore_whitespace_casing","repo":"pygongnlp/PT-M2","repo_kind":"official","path":"metrics.py","file_url":"https://github.com/pygongnlp/PT-M2/blob/HEAD/metrics.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"98662233319d385a"}},{"code_sha256_prefix":"10b705f17bd01668","entry":"get_distance","repo":"pygongnlp/PT-M2","repo_kind":"official","path":"metrics.py","file_url":"https://github.com/pygongnlp/PT-M2/blob/HEAD/metrics.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"10b705f17bd01668"}},{"code_sha256_prefix":"41ef3246fa487f1d","entry":"get_ref","repo":"pygongnlp/PT-M2","repo_kind":"official","path":"metrics.py","file_url":"https://github.com/pygongnlp/PT-M2/blob/HEAD/metrics.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"41ef3246fa487f1d"}},{"code_sha256_prefix":"b6ac128b4b5ce1c0","entry":"levenshtein_matrix","repo":"pygongnlp/PT-M2","repo_kind":"official","path":"metrics.py","file_url":"https://github.com/pygongnlp/PT-M2/blob/HEAD/metrics.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"b6ac128b4b5ce1c0"}},{"code_sha256_prefix":"b8a21170214bed29","entry":"load_annotation","repo":"pygongnlp/PT-M2","repo_kind":"official","path":"metrics.py","file_url":"https://github.com/pygongnlp/PT-M2/blob/HEAD/metrics.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"b8a21170214bed29"}},{"code_sha256_prefix":"132e8a705869c3fa","entry":"matchEdit","repo":"pygongnlp/PT-M2","repo_kind":"official","path":"metrics.py","file_url":"https://github.com/pygongnlp/PT-M2/blob/HEAD/metrics.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"132e8a705869c3fa"}},{"code_sha256_prefix":"83246c9a671abeec","entry":"matchSeq","repo":"pygongnlp/PT-M2","repo_kind":"official","path":"metrics.py","file_url":"https://github.com/pygongnlp/PT-M2/blob/HEAD/metrics.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"83246c9a671abeec"}},{"code_sha256_prefix":"74c31a434f3b4be0","entry":"merge_edits","repo":"pygongnlp/PT-M2","repo_kind":"official","path":"metrics.py","file_url":"https://github.com/pygongnlp/PT-M2/blob/HEAD/metrics.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"74c31a434f3b4be0"}},{"code_sha256_prefix":"875d2b0093f14f77","entry":"merge_graph","repo":"pygongnlp/PT-M2","repo_kind":"official","path":"metrics.py","file_url":"https://github.com/pygongnlp/PT-M2/blob/HEAD/metrics.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"875d2b0093f14f77"}},{"code_sha256_prefix":"a023f2f843d836b2","entry":"paragraphs","repo":"pygongnlp/PT-M2","repo_kind":"official","path":"metrics.py","file_url":"https://github.com/pygongnlp/PT-M2/blob/HEAD/metrics.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"a023f2f843d836b2"}},{"code_sha256_prefix":"e8cf8b92ebfb0e39","entry":"process_edits","repo":"pygongnlp/PT-M2","repo_kind":"official","path":"metrics.py","file_url":"https://github.com/pygongnlp/PT-M2/blob/HEAD/metrics.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"e8cf8b92ebfb0e39"}},{"code_sha256_prefix":"623c7d0f98224e0f","entry":"set_weights","repo":"pygongnlp/PT-M2","repo_kind":"official","path":"metrics.py","file_url":"https://github.com/pygongnlp/PT-M2/blob/HEAD/metrics.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"623c7d0f98224e0f"}},{"code_sha256_prefix":"b122957232c466b9","entry":"shrinkEdit","repo":"pygongnlp/PT-M2","repo_kind":"official","path":"metrics.py","file_url":"https://github.com/pygongnlp/PT-M2/blob/HEAD/metrics.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"b122957232c466b9"}},{"code_sha256_prefix":"a45052c27ec03581","entry":"simplify_edits","repo":"pygongnlp/PT-M2","repo_kind":"official","path":"metrics.py","file_url":"https://github.com/pygongnlp/PT-M2/blob/HEAD/metrics.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"a45052c27ec03581"}},{"code_sha256_prefix":"2cfd71ded15bd59d","entry":"smart_open","repo":"pygongnlp/PT-M2","repo_kind":"official","path":"metrics.py","file_url":"https://github.com/pygongnlp/PT-M2/blob/HEAD/metrics.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"2cfd71ded15bd59d"}},{"code_sha256_prefix":"fac4285ee68be04f","entry":"transitive_arcs","repo":"pygongnlp/PT-M2","repo_kind":"official","path":"metrics.py","file_url":"https://github.com/pygongnlp/PT-M2/blob/HEAD/metrics.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"fac4285ee68be04f"}},{"code_sha256_prefix":"a131873a0a2572e5","entry":"BARTScorer","repo":"pygongnlp/PT-M2","repo_kind":"official","path":"metrics.py","file_url":"https://github.com/pygongnlp/PT-M2/blob/HEAD/metrics.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"a131873a0a2572e5"}},{"code_sha256_prefix":"87624a3984fe1708","entry":"PTM2","repo":"pygongnlp/PT-M2","repo_kind":"official","path":"metrics.py","file_url":"https://github.com/pygongnlp/PT-M2/blob/HEAD/metrics.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"87624a3984fe1708"}},{"code_sha256_prefix":"84908af1d51cf18c","entry":"batch_multi_pre_rec_f1","repo":"pygongnlp/PT-M2","repo_kind":"official","path":"metrics.py","file_url":"https://github.com/pygongnlp/PT-M2/blob/HEAD/metrics.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"84908af1d51cf18c"}},{"code_sha256_prefix":"eb3b71d268d7c15f","entry":"batch_multi_pre_rec_f1_errant","repo":"pygongnlp/PT-M2","repo_kind":"official","path":"metrics.py","file_url":"https://github.com/pygongnlp/PT-M2/blob/HEAD/metrics.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"eb3b71d268d7c15f"}},{"code_sha256_prefix":"320d76c801f61bae","entry":"batch_multi_pre_rec_f1_sent","repo":"pygongnlp/PT-M2","repo_kind":"official","path":"metrics.py","file_url":"https://github.com/pygongnlp/PT-M2/blob/HEAD/metrics.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"320d76c801f61bae"}},{"code_sha256_prefix":"35662847bc7bac1f","entry":"batch_multi_pre_rec_f1_sent_errant","repo":"pygongnlp/PT-M2","repo_kind":"official","path":"metrics.py","file_url":"https://github.com/pygongnlp/PT-M2/blob/HEAD/metrics.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"35662847bc7bac1f"}},{"code_sha256_prefix":"0b32a42c69fe0b38","entry":"compute_weight_edits","repo":"pygongnlp/PT-M2","repo_kind":"official","path":"metrics.py","file_url":"https://github.com/pygongnlp/PT-M2/blob/HEAD/metrics.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"0b32a42c69fe0b38"}},{"code_sha256_prefix":"e3c94f36e8f91c28","entry":"errant_load_annotation","repo":"pygongnlp/PT-M2","repo_kind":"official","path":"metrics.py","file_url":"https://github.com/pygongnlp/PT-M2/blob/HEAD/metrics.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"e3c94f36e8f91c28"}},{"code_sha256_prefix":"b8394bc429303913","entry":"f1","repo":"pygongnlp/PT-M2","repo_kind":"official","path":"metrics.py","file_url":"https://github.com/pygongnlp/PT-M2/blob/HEAD/metrics.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"b8394bc429303913"}},{"code_sha256_prefix":"2c259600beaaa31b","entry":"pre_rec_f1","repo":"pygongnlp/PT-M2","repo_kind":"official","path":"metrics.py","file_url":"https://github.com/pygongnlp/PT-M2/blob/HEAD/metrics.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"2c259600beaaa31b"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}