{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/lucid-llm-generated-utterances-for-complex","title":"LUCID: LLM-Generated Utterances for Complex and Interesting Dialogues","arxiv_id":"2403.00462","date":"2024-03-01","proceeding":null,"authors":["Joe Stacey","Jianpeng Cheng","John Torr","Tristan Guigue","Joris Driesen","Alexandru Coca","Mark Gaynor","Anders Johannsen"],"abstract":"Spurred by recent advances in Large Language Models (LLMs), virtual assistants are poised to take a leap forward in terms of their dialogue capabilities. Yet a major bottleneck to achieving genuinely transformative task-oriented dialogue capabilities remains the scarcity of high quality data. Existing datasets, while impressive in scale, have limited domain coverage and contain few genuinely challenging conversational phenomena; those which are present are typically unlabelled, making it difficult to assess the strengths and weaknesses of models without time-consuming and costly human evaluation. Moreover, creating high quality dialogue data has until now required considerable human input, limiting both the scale of these datasets and the ability to rapidly bootstrap data for a new target domain. We aim to overcome these issues with LUCID, a modularised and highly automated LLM-driven data generation system that produces realistic, diverse and challenging dialogues. We use LUCID to generate a seed dataset of 4,277 conversations across 100 intents to demonstrate its capabilities, with a human review finding consistently high quality labels in the generated data.","url_abs":"https://arxiv.org/abs/2403.00462v2","url_pdf":"https://arxiv.org/pdf/2403.00462v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"lucid-llm-generated-utterances-for-complex","repo_url":"https://github.com/apple/ml-lucid-datagen","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok","spdx":"NOASSERTION"}}],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2403.00462","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2403.00462"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/apple/ml-lucid-datagen","reach":{"status":"ok","spdx":"NOASSERTION"}}],"summary":{"ran":4,"unverified":3},"by_repo_kind":{"official":{"samples":7,"ran":4,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":7,"samples":[{"code_sha256_prefix":"c9eb161a9e36a404","entry":"check_str_responses_said_by_user","repo":"apple/ml-lucid-datagen","repo_kind":"official","path":"lucid_generate_data/generate_str_slot_values.py","file_url":"https://github.com/apple/ml-lucid-datagen/blob/HEAD/lucid_generate_data/generate_str_slot_values.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"c9eb161a9e36a404"}},{"code_sha256_prefix":"8364d8da800ffdc7","entry":"extract_conversation","repo":"apple/ml-lucid-datagen","repo_kind":"official","path":"lucid_generate_data/compile_data.py","file_url":"https://github.com/apple/ml-lucid-datagen/blob/HEAD/lucid_generate_data/compile_data.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"8364d8da800ffdc7"}},{"code_sha256_prefix":"9396b794aac3ed5a","entry":"is_valid_user_turn","repo":"apple/ml-lucid-datagen","repo_kind":"official","path":"lucid_generate_data/generate_user_turn.py","file_url":"https://github.com/apple/ml-lucid-datagen/blob/HEAD/lucid_generate_data/generate_user_turn.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"9396b794aac3ed5a"}},{"code_sha256_prefix":"09d339d9bba6217e","entry":"ref_last_hint_only","repo":"apple/ml-lucid-datagen","repo_kind":"official","path":"lucid_generate_data/generate_system_turn.py","file_url":"https://github.com/apple/ml-lucid-datagen/blob/HEAD/lucid_generate_data/generate_system_turn.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"09d339d9bba6217e"}},{"code_sha256_prefix":"e253bff10c5b3fd5","entry":"add_select_system_tags","repo":"apple/ml-lucid-datagen","repo_kind":"official","path":"lucid_generate_data/utils_compile_data.py","file_url":"https://github.com/apple/ml-lucid-datagen/blob/HEAD/lucid_generate_data/utils_compile_data.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"e253bff10c5b3fd5"}},{"code_sha256_prefix":"556c27b3b0e62a70","entry":"save_validation_results","repo":"apple/ml-lucid-datagen","repo_kind":"official","path":"lucid_generate_data/generate_system_turn.py","file_url":"https://github.com/apple/ml-lucid-datagen/blob/HEAD/lucid_generate_data/generate_system_turn.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"556c27b3b0e62a70"}},{"code_sha256_prefix":"5200d3d2f7fe903c","entry":"stage_factory","repo":"apple/ml-lucid-datagen","repo_kind":"official","path":"lucid_generate_data/stage.py","file_url":"https://github.com/apple/ml-lucid-datagen/blob/HEAD/lucid_generate_data/stage.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"5200d3d2f7fe903c"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}