{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/scaling-associative-classification-for-very","title":"Scaling associative classification for very large datasets","arxiv_id":"1805.03887","date":"2018-05-10","proceeding":null,"authors":["Luca Venturini","Elena Baralis","Paolo Garza"],"abstract":"Supervised learning algorithms are nowadays successfully scaling up to\ndatasets that are very large in volume, leveraging the potential of in-memory\ncluster-computing Big Data frameworks. Still, massive datasets with a number of\nlarge-domain categorical features are a difficult challenge for any classifier.\nMost off-the-shelf solutions cannot cope with this problem. In this work we\nintroduce DAC, a Distributed Associative Classifier. DAC exploits ensemble\nlearning to distribute the training of an associative classifier among parallel\nworkers and improve the final quality of the model. Furthermore, it adopts\nseveral novel techniques to reach high scalability without sacrificing quality,\namong which a preventive pruning of classification rules in the extraction\nphase based on Gini impurity. We ran experiments on Apache Spark, on a real\nlarge-scale dataset with more than 4 billion records and 800 million distinct\ncategories. The results showed that DAC improves on a state-of-the-art solution\nin both prediction quality and execution time. Since the generated model is\nhuman-readable, it can not only classify new records, but also allow\nunderstanding both the logic behind the prediction and the properties of the\nmodel, becoming a useful aid for decision makers.","url_abs":"http://arxiv.org/abs/1805.03887v1","url_pdf":"http://arxiv.org/pdf/1805.03887v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"scaling-associative-classification-for-very","repo_url":"https://gitlab.com/dbdmg/dac","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"none","reach":null}],"tasks":[{"task_slug":"classification-1","task_name":"Classification"},{"task_slug":"ensemble-learning","task_name":"Ensemble Learning"},{"task_slug":"classification","task_name":"General Classification"}],"methods":[{"method_slug":"pruning","method_name":"Pruning"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/1805.03887","atlas_url":"https://app.syntology.ai/?focus=1805.03887","mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}