{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/relevance-of-unsupervised-metrics-in-task","title":"Relevance of Unsupervised Metrics in Task-Oriented Dialogue for Evaluating Natural Language Generation","arxiv_id":"1706.09799","date":"2017-06-29","proceeding":"ICLR 2018 1","authors":["Shikhar Sharma","Layla El Asri","Hannes Schulz","Jeremie Zumer"],"abstract":"Automated metrics such as BLEU are widely used in the machine translation\nliterature. They have also been used recently in the dialogue community for\nevaluating dialogue response generation. However, previous work in dialogue\nresponse generation has shown that these metrics do not correlate strongly with\nhuman judgment in the non task-oriented dialogue setting. Task-oriented\ndialogue responses are expressed on narrower domains and exhibit lower\ndiversity. It is thus reasonable to think that these automated metrics would\ncorrelate well with human judgment in the task-oriented setting where the\ngeneration task consists of translating dialogue acts into a sentence. We\nconduct an empirical study to confirm whether this is the case. Our findings\nindicate that these automated metrics have stronger correlation with human\njudgments in the task-oriented setting compared to what has been observed in\nthe non task-oriented setting. We also observe that these metrics correlate\neven better for datasets which provide multiple ground truth reference\nsentences. In addition, we show that some of the currently available corpora\nfor task-oriented language generation can be solved with simple models and\nadvocate for more challenging datasets.","url_abs":"http://arxiv.org/abs/1706.09799v1","url_pdf":"http://arxiv.org/pdf/1706.09799v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"relevance-of-unsupervised-metrics-in-task","repo_url":"https://github.com/Maluuba/nlg-eval","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"none","reach":null},{"paper_slug":"relevance-of-unsupervised-metrics-in-task","repo_url":"https://github.com/kingsaint/Wikidata-Descriptions","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"unanswered"}},{"paper_slug":"relevance-of-unsupervised-metrics-in-task","repo_url":"https://github.com/sonalinayak/nlg-eval","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":{"status":"unanswered"}}],"tasks":[{"task_slug":"dialogue-generation","task_name":"Dialogue Generation"},{"task_slug":"diversity","task_name":"Diversity"},{"task_slug":"machine-translation","task_name":"Machine Translation"},{"task_slug":"response-generation","task_name":"Response Generation"},{"task_slug":"sentence","task_name":"Sentence"},{"task_slug":"text-generation","task_name":"Text Generation"},{"task_slug":"translation","task_name":"Translation"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/1706.09799","atlas_url":"https://app.syntology.ai/?focus=1706.09799","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"1706.09799"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/Maluuba/nlg-eval","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/kingsaint/Wikidata-Descriptions","reach":{"status":"unanswered"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/sonalinayak/nlg-eval","reach":{"status":"unanswered"}}],"summary":{"ran_draft_wrong":2},"by_repo_kind":{"official":{"samples":2,"ran":2,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":2,"samples":[{"code_sha256_prefix":"ba9348aac71a3f72","entry":"dec","repo":"Maluuba/nlg-eval","repo_kind":"official","path":"nlgeval/pycocoevalcap/meteor/meteor.py","file_url":"https://github.com/Maluuba/nlg-eval/blob/HEAD/nlgeval/pycocoevalcap/meteor/meteor.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"ba9348aac71a3f72"}},{"code_sha256_prefix":"eaab50b5d9fea58a","entry":"enc","repo":"Maluuba/nlg-eval","repo_kind":"official","path":"nlgeval/pycocoevalcap/meteor/meteor.py","file_url":"https://github.com/Maluuba/nlg-eval/blob/HEAD/nlgeval/pycocoevalcap/meteor/meteor.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"eaab50b5d9fea58a"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}