{"url":"/dataset/dialogue-state-tracking-challenge","name":"Dialogue State Tracking Challenge","full_name":"Dialogue State Tracking Challenge","description_markdown":"The Dialog State Tracking Challenges 2 & 3 (DSTC2&3) were research challenge focused on improving the state of the art in tracking the state of spoken dialog systems. State tracking, sometimes called belief tracking, refers to accurately estimating the user's goal as a dialog progresses. Accurate state tracking is desirable because it provides robustness to errors in speech recognition, and helps reduce ambiguity inherent in language within a temporal process like dialog.\r\nIn these challenges, participants were given labelled corpora of dialogs to develop state tracking algorithms. The trackers were then evaluated on a common set of held-out dialogs, which were released, un-labelled, during a one week period.\r\n\r\nThe corpus was collected using Amazon Mechanical Turk, and consists of dialogs in two domains: restaurant information, and tourist information. Tourist information subsumes restaurant information, and includes bars, cafés etc. as well as multiple new slots. There were two rounds of evaluation using this data:\r\n\r\nDSTC 2 released a large number of training dialogs related to restaurant search. Compared to DSTC (which was in the bus timetables domain), DSTC 2 introduces changing user goals, tracking 'requested slots' as well as the new restaurants domain. Results from DSTC 2 were presented at SIGDIAL 2014.\r\nDSTC 3 addressed the problem of adaption to a new domain - tourist information. DSTC 3 releases a small amount of labelled data in the tourist information domain; participants will use this data plus the restaurant data from DSTC 2 for training.\r\nDialogs used for training are fully labelled; user transcriptions, user dialog-act semantics and dialog state are all annotated. (This corpus therefore is also suitable for studies in Spoken Language Understanding.)\r\n\r\nSource: [https://github.com/matthen/dstc](https://github.com/matthen/dstc)\r\nImage Source: [https://www.aclweb.org/anthology/W13-4065.pdf](https://www.aclweb.org/anthology/W13-4065.pdf)","description_withheld":null,"homepage":"https://github.com/matthen/dstc","introduced_date":"2013-01-01","introduced_date_note":null,"introduced_by":{"paper":"/paper/the-dialog-state-tracking-challenge","title":"The Dialog State Tracking Challenge","first_author":"Jason Williams","url":null},"license":{"name":"Unknown","url":null},"modalities":[{"name":"Texts","url":"/datasets/modality/texts"},{"name":"Dialog","url":"/datasets/modality/dialog"}],"tasks":[{"name":"Intent Detection","url":"/task/intent-detection","datasets_with_task":"/datasets/task/intent-detection"},{"name":"Deblurring","url":"/task/deblurring","datasets_with_task":"/datasets/task/deblurring"},{"name":"Visual Tracking","url":"/task/visual-tracking","datasets_with_task":"/datasets/task/visual-tracking"},{"name":"Slot Filling","url":"/task/slot-filling","datasets_with_task":"/datasets/task/slot-filling"},{"name":"Spoken Language Understanding","url":"/task/spoken-language-understanding","datasets_with_task":"/datasets/task/spoken-language-understanding"},{"name":"Dialogue State Tracking","url":"/task/dialogue-state-tracking","datasets_with_task":"/datasets/task/dialogue-state-tracking"},{"name":"domain classification","url":"/task/domain-classification","datasets_with_task":"/datasets/task/domain-classification"},{"name":"Spoken Dialogue Systems","url":"/task/spoken-dialogue-systems","datasets_with_task":"/datasets/task/spoken-dialogue-systems"}],"languages":[{"name":"English","url":"/datasets/language/english"}],"variants":["Second dialogue state tracking challenge","Dialogue State Tracking Challenge"],"data_loaders":[{"repo":"https://github.com/matthen/dstc","url":"https://github.com/matthen/dstc","frameworks":[]}],"num_papers_in_archive":33,"source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28"},"benchmarks":[{"leaderboard":"/sota/dialogue-state-tracking-on-second-dialogue","task":"Dialogue State Tracking","dataset_variant":"Second dialogue state tracking challenge","rows":7,"metrics":["Joint","Area","Food","Price","Request"],"first_row_in_archive_order":{"model":"Seq2Seq-DU-w/oSchema","paper":"/paper/a-sequence-to-sequence-approach-to-dialogue","metrics":{"Joint":"85"},"code_links":[{"title":"sweetalyssum/Seq2Seq-DU","url":"https://github.com/sweetalyssum/Seq2Seq-DU"}]},"note":"rows are the archive's own order at snapshot; nothing here re-ranks them"},{"leaderboard":"/sota/deblurring-on-second-dialogue-state-tracking","task":"Deblurring","dataset_variant":"Second dialogue state tracking challenge","rows":1,"metrics":["MAE"],"first_row_in_archive_order":{"model":"Ours","paper":"/paper/attention-stay-focus-1","metrics":{"MAE":"0.0377"},"code_links":[{"title":"tuvovan/ATTSF","url":"https://github.com/tuvovan/ATTSF"}]},"note":"rows are the archive's own order at snapshot; nothing here re-ranks them"},{"leaderboard":"/sota/intent-detection-on-dialogue-state-tracking","task":"Intent Detection","dataset_variant":"Dialogue State Tracking Challenge","rows":1,"metrics":["Accuracy"],"first_row_in_archive_order":{"model":"MIDAS","paper":"/paper/midas-multi-level-intent-domain-and-slot","metrics":{"Accuracy":"94.27"},"code_links":[{"title":"adlnlp/Midas","url":"https://github.com/adlnlp/Midas"}]},"note":"rows are the archive's own order at snapshot; nothing here re-ranks them"},{"leaderboard":"/sota/slot-filling-on-dialogue-state-tracking","task":"Slot Filling","dataset_variant":"Dialogue State Tracking Challenge","rows":1,"metrics":["F1 score"],"first_row_in_archive_order":{"model":"MIDAS","paper":"/paper/midas-multi-level-intent-domain-and-slot","metrics":{"F1 score":"98.56"},"code_links":[{"title":"adlnlp/Midas","url":"https://github.com/adlnlp/Midas"}]},"note":"rows are the archive's own order at snapshot; nothing here re-ranks them"},{"leaderboard":"/sota/visual-tracking-on-second-dialogue-state","task":"Visual Tracking","dataset_variant":"Second dialogue state tracking challenge","rows":1,"metrics":["Score"],"first_row_in_archive_order":{"model":"MDNet","paper":"/paper/learning-multi-domain-convolutional-neural","metrics":{"Score":"0.64"},"code_links":[{"title":"HyeonseobNam/py-MDNet","url":"https://github.com/HyeonseobNam/py-MDNet"},{"title":"FelixOliver/PROYECTO-FINAL-VOT","url":"https://github.com/FelixOliver/PROYECTO-FINAL-VOT"}]},"note":"rows are the archive's own order at snapshot; nothing here re-ranks them"}],"papers_with_a_benchmark_row":[{"paper":"/paper/midas-multi-level-intent-domain-and-slot","title":"MIDAS: Multi-level Intent, Domain, And Slot Knowledge Distillation for Multi-turn NLU","date":"2024-08-15","rows_on_this_dataset":2,"code_links":1,"syntology":null},{"paper":"/paper/effective-sequence-to-sequence-dialogue-state","title":"Effective Sequence-to-Sequence Dialogue State Tracking","date":"2021-08-31","rows_on_this_dataset":1,"code_links":1,"syntology":null},{"paper":"/paper/attention-stay-focus-1","title":"Attention! Stay Focus!","date":"2021-04-16","rows_on_this_dataset":1,"code_links":1,"syntology":null},{"paper":"/paper/a-sequence-to-sequence-approach-to-dialogue","title":"A Sequence-to-Sequence Approach to Dialogue State Tracking","date":"2020-11-18","rows_on_this_dataset":1,"code_links":1,"syntology":null},{"paper":"/paper/towards-universal-dialogue-state-tracking","title":"Towards Universal Dialogue State Tracking","date":"2018-10-22","rows_on_this_dataset":1,"code_links":1,"syntology":null},{"paper":"/paper/global-locally-self-attentive-dialogue-state","title":"Global-Locally Self-Attentive Dialogue State Tracker","date":"2018-05-19","rows_on_this_dataset":1,"code_links":2,"syntology":{"read_at":"2026-09-24T18:15:14+00:00","samples_harvested":8,"samples_ran":0,"samples_unverified":8,"pointer_only_for_licence":3,"claim":"Per-sample execution on synthesized fixtures; not a correctness claim."}},{"paper":"/paper/dialogue-learning-with-human-teaching-and","title":"Dialogue Learning with Human Teaching and Feedback in End-to-End Trainable Task-Oriented Dialogue Systems","date":"2018-04-18","rows_on_this_dataset":1,"code_links":1,"syntology":null},{"paper":"/paper/neural-belief-tracker-data-driven-dialogue","title":"Neural Belief Tracker: Data-Driven Dialogue State Tracking","date":"2016-06-12","rows_on_this_dataset":1,"code_links":0,"syntology":null},{"paper":"/paper/learning-multi-domain-convolutional-neural","title":"Learning Multi-Domain Convolutional Neural Networks for Visual Tracking","date":"2015-10-27","rows_on_this_dataset":1,"code_links":2,"syntology":null}],"syntology_totals":{"read_at":"2026-09-24T18:15:14+00:00","papers_with_samples":1,"samples_harvested":8,"samples_ran":0,"samples_unverified":8,"pointer_only_for_licence":3,"papers_with_no_sample_that_ran":1,"note":"the per-paper counts above, summed; not a rate"},"papers_note":"The archive never published its papers-using-dataset list; these are papers with a leaderboard row on this dataset's benchmarks."}