{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/handling-divergent-reference-texts-when","title":"Handling Divergent Reference Texts when Evaluating Table-to-Text Generation","arxiv_id":"1906.01081","date":"2019-06-03","proceeding":"ACL 2019 7","authors":["Bhuwan Dhingra","Manaal Faruqui","Ankur Parikh","Ming-Wei Chang","Dipanjan Das","William W. Cohen"],"abstract":"Automatically constructed datasets for generating text from semi-structured data (tables), such as WikiBio, often contain reference texts that diverge from the information in the corresponding semi-structured data. We show that metrics which rely solely on the reference texts, such as BLEU and ROUGE, show poor correlation with human judgments when those references diverge. We propose a new metric, PARENT, which aligns n-grams from the reference and generated texts to the semi-structured data before computing their precision and recall. Through a large scale human evaluation study of table-to-text models for WikiBio, we show that PARENT correlates with human judgments better than existing text generation metrics. We also adapt and evaluate the information extraction based evaluation proposed by Wiseman et al (2017), and show that PARENT has comparable correlation to it, while being easier to use. We show that PARENT is also applicable when the reference texts are elicited from humans using the data from the WebNLG challenge.","url_abs":"https://arxiv.org/abs/1906.01081v1","url_pdf":"https://arxiv.org/pdf/1906.01081v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"handling-divergent-reference-texts-when","repo_url":"https://github.com/KaijuML/parent","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"tf","reach":null}],"tasks":[{"task_slug":"table-to-text-generation","task_name":"Table-to-Text Generation"},{"task_slug":"text-generation","task_name":"Text Generation"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=1906.01081","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"1906.01081"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/KaijuML/parent","reach":null}],"summary":{"ran_draft_wrong":1,"ran_honours":2},"by_repo_kind":{"listed":{"samples":3,"ran":3,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":3,"samples":[{"code_sha256_prefix":"a82d9808a63b1fe2","entry":"nwise","repo":"KaijuML/parent","repo_kind":"listed","path":"parent/parent.py","file_url":"https://github.com/KaijuML/parent/blob/HEAD/parent/parent.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"a82d9808a63b1fe2"}},{"code_sha256_prefix":"a9329bb070ec61b6","entry":"overlap_probability","repo":"KaijuML/parent","repo_kind":"listed","path":"parent/parent.py","file_url":"https://github.com/KaijuML/parent/blob/HEAD/parent/parent.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"a9329bb070ec61b6"}},{"code_sha256_prefix":"eb2f1d9a66145ea8","entry":"parent_instance_level","repo":"KaijuML/parent","repo_kind":"listed","path":"parent/parent.py","file_url":"https://github.com/KaijuML/parent/blob/HEAD/parent/parent.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"eb2f1d9a66145ea8"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}