{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/a-new-evaluation-framework-for-topic-modeling","title":"A new evaluation framework for topic modeling algorithms based on synthetic corpora","arxiv_id":"1901.09848","date":"2019-01-28","proceeding":null,"authors":["Hanyu Shi","Martin Gerlach","Isabel Diersen","Doug Downey","Luis A. N. Amaral"],"abstract":"Topic models are in widespread use in natural language processing and beyond.\nHere, we propose a new framework for the evaluation of probabilistic topic\nmodeling algorithms based on synthetic corpora containing an unambiguously\ndefined ground truth topic structure. The major innovation of our approach is\nthe ability to quantify the agreement between the planted and inferred topic\nstructures by comparing the assigned topic labels at the level of the tokens.\nIn experiments, our approach yields novel insights about the relative strengths\nof topic models as corpus characteristics vary, and the first evidence of an\n\"undetectable phase\" for topic models when the planted structure is weak. We\nalso establish the practical relevance of the insights gained for synthetic\ncorpora by predicting the performance of topic modeling algorithms in\nclassification tasks in real-world corpora.","url_abs":"http://arxiv.org/abs/1901.09848v1","url_pdf":"http://arxiv.org/pdf/1901.09848v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"a-new-evaluation-framework-for-topic-modeling","repo_url":"https://github.com/amarallab/synthetic_benchmark_topic_model","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"none","reach":{"status":"ok"}}],"tasks":[{"task_slug":"classification","task_name":"General Classification"},{"task_slug":"topic-models","task_name":"Topic Models"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":null,"atlas_url":"https://app.syntology.ai/?focus=1901.09848","mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}