{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/grayscale-data-construction-and-multi-level","title":"The World is Not Binary: Learning to Rank with Grayscale Data for Dialogue Response Selection","arxiv_id":"2004.02421","date":"2020-04-06","proceeding":"EMNLP 2020 11","authors":["Zibo Lin","Deng Cai","Yan Wang","Xiaojiang Liu","Hai-Tao Zheng","Shuming Shi"],"abstract":"Response selection plays a vital role in building retrieval-based conversation systems. Despite that response selection is naturally a learning-to-rank problem, most prior works take a point-wise view and train binary classifiers for this task: each response candidate is labeled either relevant (one) or irrelevant (zero). On the one hand, this formalization can be sub-optimal due to its ignorance of the diversity of response quality. On the other hand, annotating grayscale data for learning-to-rank can be prohibitively expensive and challenging. In this work, we show that grayscale data can be automatically constructed without human effort. Our method employs off-the-shelf response retrieval models and response generation models as automatic grayscale data generators. With the constructed grayscale data, we propose multi-level ranking objectives for training, which can (1) teach a matching model to capture more fine-grained context-response relevance difference and (2) reduce the train-test discrepancy in terms of distractor strength. Our method is simple, effective, and universal. Experiments on three benchmark datasets and four state-of-the-art matching models show that the proposed approach brings significant and consistent performance improvements.","url_abs":"https://arxiv.org/abs/2004.02421v4","url_pdf":"https://arxiv.org/pdf/2004.02421v4.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[],"tasks":[{"task_slug":"conversational-response-selection","task_name":"Conversational Response Selection"},{"task_slug":"diversity","task_name":"Diversity"},{"task_slug":"learning-to-rank","task_name":"Learning-To-Rank"},{"task_slug":"response-generation","task_name":"Response Generation"},{"task_slug":"retrieval","task_name":"Retrieval"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/conversational-response-selection-on-e","task":"Conversational Response Selection","dataset":"E-commerce","model":"G-MSN","rank_in_archive_order":11,"of":15,"metrics":{"R10@1":"0.613","R10@2":"0.786","R10@5":"0.964"},"uses_additional_data":false}],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2004.02421","mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}