{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/interpretable-user-satisfaction-estimation","title":"Interpretable User Satisfaction Estimation for Conversational Systems with Large Language Models","arxiv_id":"2403.12388","date":"2024-03-19","proceeding":null,"authors":["Ying-Chun Lin","Jennifer Neville","Jack W. Stokes","Longqi Yang","Tara Safavi","Mengting Wan","Scott Counts","Siddharth Suri","Reid Andersen","Xiaofeng Xu","Deepak Gupta","Sujay Kumar Jauhar","Xia Song","Georg Buscher","Saurabh Tiwary","Brent Hecht","Jaime Teevan"],"abstract":"Accurate and interpretable user satisfaction estimation (USE) is critical for understanding, evaluating, and continuously improving conversational systems. Users express their satisfaction or dissatisfaction with diverse conversational patterns in both general-purpose (ChatGPT and Bing Copilot) and task-oriented (customer service chatbot) conversational systems. Existing approaches based on featurized ML models or text embeddings fall short in extracting generalizable patterns and are hard to interpret. In this work, we show that LLMs can extract interpretable signals of user satisfaction from their natural language utterances more effectively than embedding-based approaches. Moreover, an LLM can be tailored for USE via an iterative prompting framework using supervision from labeled examples. The resulting method, Supervised Prompting for User satisfaction Rubrics (SPUR), not only has higher accuracy but is more interpretable as it scores user satisfaction via learned rubrics with a detailed breakdown.","url_abs":"https://arxiv.org/abs/2403.12388v2","url_pdf":"https://arxiv.org/pdf/2403.12388v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[],"tasks":[{"task_slug":"chatbot","task_name":"Chatbot"}],"methods":[{"method_slug":null,"method_name":null},{"method_slug":"use","method_name":"USE"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2403.12388","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2403.12388"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/dengyang17/USDA","reach":{"status":"ok"}}],"summary":{"ran_draft_wrong":1,"ran":3,"unverified":1},"by_repo_kind":{"found_in_text":{"samples":5,"ran":4,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":5,"samples":[{"code_sha256_prefix":"8d23fbe2b99b840b","entry":"gelu","repo":"dengyang17/USDA","repo_kind":"found_in_text","path":"usda-clu/transformer.py","file_url":"https://github.com/dengyang17/USDA/blob/HEAD/usda-clu/transformer.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"8d23fbe2b99b840b"}},{"code_sha256_prefix":"28e7c08042bacaaf","entry":"read_pkl","repo":"dengyang17/USDA","repo_kind":"found_in_text","path":"usda-clu/utils.py","file_url":"https://github.com/dengyang17/USDA/blob/HEAD/usda-clu/utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"28e7c08042bacaaf"}},{"code_sha256_prefix":"8e3538aa7c7d9315","entry":"set_cuda","repo":"dengyang17/USDA","repo_kind":"found_in_text","path":"usda-clu/utils.py","file_url":"https://github.com/dengyang17/USDA/blob/HEAD/usda-clu/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"8e3538aa7c7d9315"}},{"code_sha256_prefix":"e17839223b987fa2","entry":"universal_sentence_embedding","repo":"dengyang17/USDA","repo_kind":"found_in_text","path":"usda-clu/model.py","file_url":"https://github.com/dengyang17/USDA/blob/HEAD/usda-clu/model.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"e17839223b987fa2"}},{"code_sha256_prefix":"45e30b97c6069fe7","entry":"load_data","repo":"dengyang17/USDA","repo_kind":"found_in_text","path":"usda-clu/utils.py","file_url":"https://github.com/dengyang17/USDA/blob/HEAD/usda-clu/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"45e30b97c6069fe7"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}