{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/amulet-putting-complex-multi-turn","title":"Amulet: Putting Complex Multi-Turn Conversations on the Stand with LLM Juries","arxiv_id":"2505.20451","date":"2025-05-26","proceeding":null,"authors":["Sahana Ramnath","Anurag Mudgil","Brihi Joshi","Skyler Hallinan","Xiang Ren"],"abstract":"Today, large language models are widely used as judges to evaluate responses from other language models. Hence, it is imperative to benchmark and improve these LLM-judges on real-world language model usage: a typical human-assistant conversation is lengthy, and shows significant diversity in topics, intents, and requirements across turns, e.g. social interactions, task requests, feedback. We present Amulet, a framework that leverages pertinent linguistic concepts of dialog-acts and maxims to improve the accuracy of LLM-judges on preference data with complex, multi-turn conversational context. Amulet presents valuable insights about (a) the communicative structures and intents present in the conversation (dialog acts), and (b) the satisfaction of conversational principles (maxims) by the preference responses, and uses them to make judgments. On four challenging datasets, Amulet shows that (a) humans frequently (60 to 70 percent of the time) change their intents from one turn of the conversation to the next, and (b) in 75 percent of instances, the preference responses can be differentiated via dialog acts and/or maxims, reiterating the latter's significance in judging such data. Amulet can be used either as a judge by applying the framework to a single LLM, or integrated into a jury with different LLM judges; our judges and juries show strong improvements on relevant baselines for all four datasets.","url_abs":"https://arxiv.org/abs/2505.20451v1","url_pdf":"https://arxiv.org/pdf/2505.20451v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2505.20451","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2505.20451"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/INK-USC/Amulet-DA-and-Maxim","reach":{"status":"ok"}},{"provenance":"deterministic:regex_extraction","url":"https://github.com/thunlp/ChatEval","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran":2,"unverified":1},"by_repo_kind":{"found_in_text":{"samples":3,"ran":2,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"24114c47bda02e3c","entry":"get_accuracy_calibration","repo":"thunlp/ChatEval","repo_kind":"found_in_text","path":"calc_adversarial_results_calc_score_calibration.py","file_url":"https://github.com/thunlp/ChatEval/blob/HEAD/calc_adversarial_results_calc_score_calibration.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"24114c47bda02e3c"}},{"code_sha256_prefix":"f22cc7f6755823a6","entry":"get_kappa","repo":"thunlp/ChatEval","repo_kind":"found_in_text","path":"calc_adversarial_results_calc_score_calibration.py","file_url":"https://github.com/thunlp/ChatEval/blob/HEAD/calc_adversarial_results_calc_score_calibration.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"f22cc7f6755823a6"}},{"code_sha256_prefix":"a619c795784c4043","entry":"get_conv_template","repo":"thunlp/ChatEval","repo_kind":"found_in_text","path":"FastChat/fastchat/conversation.py","file_url":"https://github.com/thunlp/ChatEval/blob/HEAD/FastChat/fastchat/conversation.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"a619c795784c4043"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}