{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/tn-eval-rubric-and-evaluation-protocols-for","title":"TN-Eval: Rubric and Evaluation Protocols for Measuring the Quality of Behavioral Therapy Notes","arxiv_id":"2503.20648","date":"2025-03-26","proceeding":null,"authors":["Raj Sanjay Shah","Lei Xu","Qianchu Liu","Jon Burnsky","Drew Bertagnolli","Chaitanya Shivade"],"abstract":"Behavioral therapy notes are important for both legal compliance and patient care. Unlike progress notes in physical health, quality standards for behavioral therapy notes remain underdeveloped. To address this gap, we collaborated with licensed therapists to design a comprehensive rubric for evaluating therapy notes across key dimensions: completeness, conciseness, and faithfulness. Further, we extend a public dataset of behavioral health conversations with therapist-written notes and LLM-generated notes, and apply our evaluation framework to measure their quality. We find that: (1) A rubric-based manual evaluation protocol offers more reliable and interpretable results than traditional Likert-scale annotations. (2) LLMs can mimic human evaluators in assessing completeness and conciseness but struggle with faithfulness. (3) Therapist-written notes often lack completeness and conciseness, while LLM-generated notes contain hallucination. Surprisingly, in a blind test, therapists prefer and judge LLM-generated notes to be superior to therapist-written notes.","url_abs":"https://arxiv.org/abs/2503.20648v1","url_pdf":"https://arxiv.org/pdf/2503.20648v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[],"tasks":[{"task_slug":"hallucination","task_name":"Hallucination"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2503.20648","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2503.20648"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/amazon-science/TN-Eval","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"unverified":6},"by_repo_kind":{"found_in_text":{"samples":6,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"32e310c43fefe07d","entry":"format_cache_key","repo":"amazon-science/TN-Eval","repo_kind":"found_in_text","path":"src/run_metrics_reference_free.py","file_url":"https://github.com/amazon-science/TN-Eval/blob/HEAD/src/run_metrics_reference_free.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"32e310c43fefe07d"}},{"code_sha256_prefix":"e489c799c3460611","entry":"get_bedrock_session","repo":"amazon-science/TN-Eval","repo_kind":"found_in_text","path":"src/generate_soap_note.py","file_url":"https://github.com/amazon-science/TN-Eval/blob/HEAD/src/generate_soap_note.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"e489c799c3460611"}},{"code_sha256_prefix":"d8bd46ebfc814e0b","entry":"parse_likert","repo":"amazon-science/TN-Eval","repo_kind":"found_in_text","path":"src/run_metrics_reference_free.py","file_url":"https://github.com/amazon-science/TN-Eval/blob/HEAD/src/run_metrics_reference_free.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"d8bd46ebfc814e0b"}},{"code_sha256_prefix":"70be1bae7ab12c5d","entry":"parse_yes_no","repo":"amazon-science/TN-Eval","repo_kind":"found_in_text","path":"src/run_metrics_reference_free.py","file_url":"https://github.com/amazon-science/TN-Eval/blob/HEAD/src/run_metrics_reference_free.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"70be1bae7ab12c5d"}},{"code_sha256_prefix":"d1e65b6e8a0945c7","entry":"predict_claude3","repo":"amazon-science/TN-Eval","repo_kind":"found_in_text","path":"src/generate_soap_note.py","file_url":"https://github.com/amazon-science/TN-Eval/blob/HEAD/src/generate_soap_note.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"d1e65b6e8a0945c7"}},{"code_sha256_prefix":"45226b5558cfe05e","entry":"predict_llama3","repo":"amazon-science/TN-Eval","repo_kind":"found_in_text","path":"src/generate_soap_note.py","file_url":"https://github.com/amazon-science/TN-Eval/blob/HEAD/src/generate_soap_note.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"45226b5558cfe05e"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}