{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/designing-a-dashboard-for-transparency-and","title":"Designing a Dashboard for Transparency and Control of Conversational AI","arxiv_id":"2406.07882","date":"2024-06-12","proceeding":null,"authors":["Yida Chen","Aoyu Wu","Trevor DePodesta","Catherine Yeh","Kenneth Li","Nicholas Castillo Marin","Oam Patel","Jan Riecke","Shivam Raval","Olivia Seow","Martin Wattenberg","Fernanda Viégas"],"abstract":"Conversational LLMs function as black box systems, leaving users guessing about why they see the output they do. This lack of transparency is potentially problematic, especially given concerns around bias and truthfulness. To address this issue, we present an end-to-end prototype-connecting interpretability techniques with user experience design-that seeks to make chatbots more transparent. We begin by showing evidence that a prominent open-source LLM has a \"user model\": examining the internal state of the system, we can extract data related to a user's age, gender, educational level, and socioeconomic status. Next, we describe the design of a dashboard that accompanies the chatbot interface, displaying this user model in real time. The dashboard can also be used to control the user model and the system's behavior. Finally, we discuss a study in which users conversed with the instrumented system. Our results suggest that users appreciate seeing internal states, which helped them expose biased behavior and increased their sense of control. Participants also made valuable suggestions that point to future directions for both design and machine learning research. The project page and video demo of our TalkTuner system are available at https://bit.ly/talktuner-project-page","url_abs":"https://arxiv.org/abs/2406.07882v3","url_pdf":"https://arxiv.org/pdf/2406.07882v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"designing-a-dashboard-for-transparency-and","repo_url":"https://github.com/yc015/talktuner-chatbot-llm-dashboard","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"chatbot","task_name":"Chatbot"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2406.07882","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2406.07882"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/yc015/talktuner-chatbot-llm-dashboard","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":21,"unverified":4},"by_repo_kind":{"official":{"samples":25,"ran":21,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"c65af88d2f239ac4","entry":"async_post_service","repo":"yc015/talktuner-chatbot-llm-dashboard","repo_kind":"official","path":"dashboard_v1/dashboard_task_queue/worker/worker.py","file_url":"https://github.com/yc015/talktuner-chatbot-llm-dashboard/blob/HEAD/dashboard_v1/dashboard_task_queue/worker/worker.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c65af88d2f239ac4"}},{"code_sha256_prefix":"ddba7773fd3d51f7","entry":"exp_evidence","repo":"yc015/talktuner-chatbot-llm-dashboard","repo_kind":"official","path":"src/losses.py","file_url":"https://github.com/yc015/talktuner-chatbot-llm-dashboard/blob/HEAD/src/losses.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ddba7773fd3d51f7"}},{"code_sha256_prefix":"dd58a8d833cc8b6c","entry":"extract_activations_control","repo":"yc015/talktuner-chatbot-llm-dashboard","repo_kind":"official","path":"dashboard_v1/probing/train_probes.py","file_url":"https://github.com/yc015/talktuner-chatbot-llm-dashboard/blob/HEAD/dashboard_v1/probing/train_probes.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"dd58a8d833cc8b6c"}},{"code_sha256_prefix":"c9161e4fd0894771","entry":"format_conversation_for_model","repo":"yc015/talktuner-chatbot-llm-dashboard","repo_kind":"official","path":"dashboard_v1/probing/train_probes.py","file_url":"https://github.com/yc015/talktuner-chatbot-llm-dashboard/blob/HEAD/dashboard_v1/probing/train_probes.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c9161e4fd0894771"}},{"code_sha256_prefix":"5d58ce568acbe617","entry":"generate_conversation","repo":"yc015/talktuner-chatbot-llm-dashboard","repo_kind":"official","path":"dashboard_v1/probing/generate_conversation_dataset.py","file_url":"https://github.com/yc015/talktuner-chatbot-llm-dashboard/blob/HEAD/dashboard_v1/probing/generate_conversation_dataset.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"5d58ce568acbe617"}},{"code_sha256_prefix":"bc6d4f97d42f9501","entry":"generate_dataset","repo":"yc015/talktuner-chatbot-llm-dashboard","repo_kind":"official","path":"dashboard_v1/probing/generate_conversation_dataset.py","file_url":"https://github.com/yc015/talktuner-chatbot-llm-dashboard/blob/HEAD/dashboard_v1/probing/generate_conversation_dataset.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"bc6d4f97d42f9501"}},{"code_sha256_prefix":"79a8e24307cba632","entry":"list_available_datasets","repo":"yc015/talktuner-chatbot-llm-dashboard","repo_kind":"official","path":"dashboard_v1/probing/inspect_dataset.py","file_url":"https://github.com/yc015/talktuner-chatbot-llm-dashboard/blob/HEAD/dashboard_v1/probing/inspect_dataset.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"79a8e24307cba632"}},{"code_sha256_prefix":"a8ab0f29b2f5cd13","entry":"llama_v2_prompt","repo":"yc015/talktuner-chatbot-llm-dashboard","repo_kind":"official","path":"src/dataset.py","file_url":"https://github.com/yc015/talktuner-chatbot-llm-dashboard/blob/HEAD/src/dataset.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a8ab0f29b2f5cd13"}},{"code_sha256_prefix":"01f8e1cc8000adb6","entry":"llama_v2_reverse","repo":"yc015/talktuner-chatbot-llm-dashboard","repo_kind":"official","path":"src/prompt_utils.py","file_url":"https://github.com/yc015/talktuner-chatbot-llm-dashboard/blob/HEAD/src/prompt_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"01f8e1cc8000adb6"}},{"code_sha256_prefix":"01ff948df86446e4","entry":"load_dataset","repo":"yc015/talktuner-chatbot-llm-dashboard","repo_kind":"official","path":"dashboard_v1/probing/inspect_dataset.py","file_url":"https://github.com/yc015/talktuner-chatbot-llm-dashboard/blob/HEAD/dashboard_v1/probing/inspect_dataset.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"01ff948df86446e4"}},{"code_sha256_prefix":"afc1a18cc3093f96","entry":"load_probe_classifier","repo":"yc015/talktuner-chatbot-llm-dashboard","repo_kind":"official","path":"src/intervention_utils.py","file_url":"https://github.com/yc015/talktuner-chatbot-llm-dashboard/blob/HEAD/src/intervention_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"afc1a18cc3093f96"}},{"code_sha256_prefix":"d018404cd06ecc3f","entry":"one_hot_embedding","repo":"yc015/talktuner-chatbot-llm-dashboard","repo_kind":"official","path":"src/losses.py","file_url":"https://github.com/yc015/talktuner-chatbot-llm-dashboard/blob/HEAD/src/losses.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d018404cd06ecc3f"}},{"code_sha256_prefix":"be819d515c7481ad","entry":"process_chat","repo":"yc015/talktuner-chatbot-llm-dashboard","repo_kind":"official","path":"dashboard_v1/dashboard_task_queue/worker/worker.py","file_url":"https://github.com/yc015/talktuner-chatbot-llm-dashboard/blob/HEAD/dashboard_v1/dashboard_task_queue/worker/worker.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"be819d515c7481ad"}},{"code_sha256_prefix":"d6490eaa408ffc13","entry":"process_chat_batched","repo":"yc015/talktuner-chatbot-llm-dashboard","repo_kind":"official","path":"dashboard_v1/dashboard_task_queue/worker/worker.py","file_url":"https://github.com/yc015/talktuner-chatbot-llm-dashboard/blob/HEAD/dashboard_v1/dashboard_task_queue/worker/worker.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d6490eaa408ffc13"}},{"code_sha256_prefix":"efecc687b786a0cc","entry":"relu_evidence","repo":"yc015/talktuner-chatbot-llm-dashboard","repo_kind":"official","path":"src/losses.py","file_url":"https://github.com/yc015/talktuner-chatbot-llm-dashboard/blob/HEAD/src/losses.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"efecc687b786a0cc"}},{"code_sha256_prefix":"c90fd1d879571fb6","entry":"remove_last_k_words","repo":"yc015/talktuner-chatbot-llm-dashboard","repo_kind":"official","path":"src/dataset.py","file_url":"https://github.com/yc015/talktuner-chatbot-llm-dashboard/blob/HEAD/src/dataset.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c90fd1d879571fb6"}},{"code_sha256_prefix":"ed57d60ed6258a97","entry":"return_classifier_dict","repo":"yc015/talktuner-chatbot-llm-dashboard","repo_kind":"official","path":"src/intervention_utils.py","file_url":"https://github.com/yc015/talktuner-chatbot-llm-dashboard/blob/HEAD/src/intervention_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ed57d60ed6258a97"}},{"code_sha256_prefix":"764ffaea8c47e018","entry":"split_conversation","repo":"yc015/talktuner-chatbot-llm-dashboard","repo_kind":"official","path":"src/dataset.py","file_url":"https://github.com/yc015/talktuner-chatbot-llm-dashboard/blob/HEAD/src/dataset.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"764ffaea8c47e018"}},{"code_sha256_prefix":"a6afc594b6f4973c","entry":"split_into_messages","repo":"yc015/talktuner-chatbot-llm-dashboard","repo_kind":"official","path":"src/intervention_utils.py","file_url":"https://github.com/yc015/talktuner-chatbot-llm-dashboard/blob/HEAD/src/intervention_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a6afc594b6f4973c"}},{"code_sha256_prefix":"5d1a14b83de26cdc","entry":"start_probe_training","repo":"yc015/talktuner-chatbot-llm-dashboard","repo_kind":"official","path":"dashboard_v1/probing/client_example.py","file_url":"https://github.com/yc015/talktuner-chatbot-llm-dashboard/blob/HEAD/dashboard_v1/probing/client_example.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"5d1a14b83de26cdc"}},{"code_sha256_prefix":"84df9b0e3fc9b985","entry":"validate_dataset","repo":"yc015/talktuner-chatbot-llm-dashboard","repo_kind":"official","path":"dashboard_v1/probing/inspect_dataset.py","file_url":"https://github.com/yc015/talktuner-chatbot-llm-dashboard/blob/HEAD/dashboard_v1/probing/inspect_dataset.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"84df9b0e3fc9b985"}},{"code_sha256_prefix":"93c779c76b35a006","entry":"get_task_status","repo":"yc015/talktuner-chatbot-llm-dashboard","repo_kind":"official","path":"dashboard_v1/probing/client_example.py","file_url":"https://github.com/yc015/talktuner-chatbot-llm-dashboard/blob/HEAD/dashboard_v1/probing/client_example.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"93c779c76b35a006"}},{"code_sha256_prefix":"d0a8ad2ed431e1b5","entry":"load_model_and_tokenizer","repo":"yc015/talktuner-chatbot-llm-dashboard","repo_kind":"official","path":"dashboard_v1/probing/train_probes.py","file_url":"https://github.com/yc015/talktuner-chatbot-llm-dashboard/blob/HEAD/dashboard_v1/probing/train_probes.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d0a8ad2ed431e1b5"}},{"code_sha256_prefix":"73316eb27afceef3","entry":"save_dataset","repo":"yc015/talktuner-chatbot-llm-dashboard","repo_kind":"official","path":"dashboard_v1/probing/generate_conversation_dataset.py","file_url":"https://github.com/yc015/talktuner-chatbot-llm-dashboard/blob/HEAD/dashboard_v1/probing/generate_conversation_dataset.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"73316eb27afceef3"}},{"code_sha256_prefix":"e12290fa324335e9","entry":"wait_for_task_completion","repo":"yc015/talktuner-chatbot-llm-dashboard","repo_kind":"official","path":"dashboard_v1/probing/client_example.py","file_url":"https://github.com/yc015/talktuner-chatbot-llm-dashboard/blob/HEAD/dashboard_v1/probing/client_example.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e12290fa324335e9"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}