{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/rade-reference-assisted-dialogue-evaluation","title":"RADE: Reference-Assisted Dialogue Evaluation for Open-Domain Dialogue","arxiv_id":"2309.08156","date":"2023-09-15","proceeding":null,"authors":["Zhengliang Shi","Weiwei Sun","Shuo Zhang","Zhen Zhang","Pengjie Ren","Zhaochun Ren"],"abstract":"Evaluating open-domain dialogue systems is challenging for reasons such as the one-to-many problem, i.e., many appropriate responses other than just the golden response. As of now, automatic evaluation methods need better consistency with humans, while reliable human evaluation can be time- and cost-intensive. To this end, we propose the Reference-Assisted Dialogue Evaluation (RADE) approach under the multi-task learning framework, which leverages the pre-created utterance as reference other than the gold response to relief the one-to-many problem. Specifically, RADE explicitly compares reference and the candidate response to predict their overall scores. Moreover, an auxiliary response generation task enhances prediction via a shared encoder. To support RADE, we extend three datasets with additional rated responses other than just a golden response by human annotation. Experiments on our three datasets and two existing benchmarks demonstrate the effectiveness of our method, where Pearson, Spearman, and Kendall correlations with human evaluation outperform state-of-the-art baselines.","url_abs":"https://arxiv.org/abs/2309.08156v2","url_pdf":"https://arxiv.org/pdf/2309.08156v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[],"tasks":[{"task_slug":"dialogue-evaluation","task_name":"Dialogue Evaluation"},{"task_slug":"multi-task-learning","task_name":"Multi-Task Learning"},{"task_slug":"response-generation","task_name":"Response Generation"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2309.08156","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2309.08156"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/e0397123/dstc10_metric_track","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran":1,"unverified":1},"by_repo_kind":{"found_in_text":{"samples":2,"ran":1,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"8185af1c0cccfd8c","entry":"compute_fm_score","repo":"e0397123/dstc10_metric_track","repo_kind":"found_in_text","path":"baselines/deep_amfm/compute_dial.py","file_url":"https://github.com/e0397123/dstc10_metric_track/blob/HEAD/baselines/deep_amfm/compute_dial.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"8185af1c0cccfd8c"}},{"code_sha256_prefix":"da07c8035275a120","entry":"normalize_df","repo":"e0397123/dstc10_metric_track","repo_kind":"found_in_text","path":"baselines/deep_amfm/compute_dial.py","file_url":"https://github.com/e0397123/dstc10_metric_track/blob/HEAD/baselines/deep_amfm/compute_dial.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"da07c8035275a120"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}