{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/deconstruct-to-reconstruct-a-configurable","title":"Deconstruct to Reconstruct a Configurable Evaluation Metric for Open-Domain Dialogue Systems","arxiv_id":"2011.00483","date":"2020-11-01","proceeding":"COLING 2020 8","authors":["Vitou Phy","Yang Zhao","Akiko Aizawa"],"abstract":"Many automatic evaluation metrics have been proposed to score the overall quality of a response in open-domain dialogue. Generally, the overall quality is comprised of various aspects, such as relevancy, specificity, and empathy, and the importance of each aspect differs according to the task. For instance, specificity is mandatory in a food-ordering dialogue task, whereas fluency is preferred in a language-teaching dialogue system. However, existing metrics are not designed to cope with such flexibility. For example, BLEU score fundamentally relies only on word overlapping, whereas BERTScore relies on semantic similarity between reference and candidate response. Thus, they are not guaranteed to capture the required aspects, i.e., specificity. To design a metric that is flexible to a task, we first propose making these qualities manageable by grouping them into three groups: understandability, sensibleness, and likability, where likability is a combination of qualities that are essential for a task. We also propose a simple method to composite metrics of each aspect to obtain a single metric called USL-H, which stands for Understandability, Sensibleness, and Likability in Hierarchy. We demonstrated that USL-H score achieves good correlations with human judgment and maintains its configurability towards different aspects and metrics.","url_abs":"https://arxiv.org/abs/2011.00483v1","url_pdf":"https://arxiv.org/pdf/2011.00483v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"deconstruct-to-reconstruct-a-configurable","repo_url":"https://github.com/vitouphy/usl_dialogue_metric","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"dialogue-evaluation","task_name":"Dialogue Evaluation"},{"task_slug":"semantic-similarity","task_name":"Semantic Similarity"},{"task_slug":"semantic-textual-similarity","task_name":"Semantic Textual Similarity"},{"task_slug":"specificity","task_name":"Specificity"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2011.00483","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2011.00483"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/vitouphy/usl_dialogue_metric","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"unverified":7},"by_repo_kind":{"official":{"samples":7,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"41a79d03a37cc6b8","entry":"apply_word_drop","repo":"vitouphy/usl_dialogue_metric","repo_kind":"official","path":"usl_score/data_utils.py","file_url":"https://github.com/vitouphy/usl_dialogue_metric/blob/HEAD/usl_score/data_utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"41a79d03a37cc6b8"}},{"code_sha256_prefix":"6dff05995d96e41f","entry":"apply_word_order","repo":"vitouphy/usl_dialogue_metric","repo_kind":"official","path":"usl_score/data_utils.py","file_url":"https://github.com/vitouphy/usl_dialogue_metric/blob/HEAD/usl_score/data_utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"6dff05995d96e41f"}},{"code_sha256_prefix":"776969e513361b74","entry":"calc_minmax","repo":"vitouphy/usl_dialogue_metric","repo_kind":"official","path":"usl_score/normalize.py","file_url":"https://github.com/vitouphy/usl_dialogue_metric/blob/HEAD/usl_score/normalize.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"776969e513361b74"}},{"code_sha256_prefix":"8e7771cd0a0658c0","entry":"composite","repo":"vitouphy/usl_dialogue_metric","repo_kind":"official","path":"usl_score/models/composite.py","file_url":"https://github.com/vitouphy/usl_dialogue_metric/blob/HEAD/usl_score/models/composite.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"8e7771cd0a0658c0"}},{"code_sha256_prefix":"2efc8ad4fb7aac77","entry":"composite_one_instance","repo":"vitouphy/usl_dialogue_metric","repo_kind":"official","path":"usl_score/models/composite.py","file_url":"https://github.com/vitouphy/usl_dialogue_metric/blob/HEAD/usl_score/models/composite.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"2efc8ad4fb7aac77"}},{"code_sha256_prefix":"a2a250c29b16791f","entry":"distinct","repo":"vitouphy/usl_dialogue_metric","repo_kind":"official","path":"usl_score/models/distinct.py","file_url":"https://github.com/vitouphy/usl_dialogue_metric/blob/HEAD/usl_score/models/distinct.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a2a250c29b16791f"}},{"code_sha256_prefix":"f76a87f857f7b328","entry":"read_dataset","repo":"vitouphy/usl_dialogue_metric","repo_kind":"official","path":"usl_score/data_utils.py","file_url":"https://github.com/vitouphy/usl_dialogue_metric/blob/HEAD/usl_score/data_utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f76a87f857f7b328"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}