{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/using-simpsons-paradox-to-discover","title":"Using Simpson's Paradox to Discover Interesting Patterns in Behavioral Data","arxiv_id":"1805.03094","date":"2018-05-08","proceeding":null,"authors":["Nazanin Alipourfard","Peter G. Fennell","Kristina Lerman"],"abstract":"We describe a data-driven discovery method that leverages Simpson's paradox\nto uncover interesting patterns in behavioral data. Our method systematically\ndisaggregates data to identify subgroups within a population whose behavior\ndeviates significantly from the rest of the population. Given an outcome of\ninterest and a set of covariates, the method follows three steps. First, it\ndisaggregates data into subgroups, by conditioning on a particular covariate,\nso as minimize the variation of the outcome within the subgroups. Next, it\nmodels the outcome as a linear function of another covariate, both in the\nsubgroups and in the aggregate data. Finally, it compares trends to identify\ndisaggregations that produce subgroups with different behaviors from the\naggregate. We illustrate the method by applying it to three real-world\nbehavioral datasets, including Q\\&A site Stack Exchange and online learning\nplatforms Khan Academy and Duolingo.","url_abs":"http://arxiv.org/abs/1805.03094v1","url_pdf":"http://arxiv.org/pdf/1805.03094v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"using-simpsons-paradox-to-discover","repo_url":"https://github.com/ninoch/Trend-Simpsons-Paradox","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok"}}],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=1805.03094","mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}