{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/deduplication-over-heterogeneous-attribute","title":"Deduplication Over Heterogeneous Attribute Types (D-HAT)","arxiv_id":null,"date":"2022-11-24","proceeding":"Advanced Data Mining and Applications (ADMA) 2022 11","authors":["Loujain Liekah","George Papadakis"],"abstract":"Deduplication is the task of recognizing multiple representations of the same real-world object. The majority of existing solutions focuses on textual data, this means that data sets containing boolean and numerical attribute types are rarely considered in the literature, while the problem of missing values is inadequately covered. Supervised solutions cannot be applied without an adequate number of labelled examples, but training data for deduplication can only be obtained through time-costly processes. In high dimensional data sets, feature engineering is also required to avoid the risk of overfitting. To address these challenges, we go beyond existing works through D-HAT, a clustering-based pipeline that is inherently capable of handling high dimensional, sparse and heterogeneous attribute types. At its core lies: (i) a novel matching function that effectively summarizes multiple matching signals, and (ii) MutMax, a greedy clustering algorithm that designates as duplicates the pairs with a mutually maximum matching score. We evaluate D-HAT on five established, real-world benchmark data sets, demonstrating that our approach outperforms the state-of-the-art supervised and unsupervised deduplication algorithms to a significant extent.","url_abs":"https://link.springer.com/chapter/10.1007/978-3-031-22137-8_28","url_pdf":"https://www.researchgate.net/publication/365673371_Deduplication_Over_Heterogeneous_Attribute_Types_D-HAT","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"deduplication-over-heterogeneous-attribute","repo_url":"https://github.com/Loujainl/D-HAT","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"none","reach":null}],"tasks":[{"task_slug":"attribute","task_name":"Attribute"},{"task_slug":"clustering","task_name":"Clustering"},{"task_slug":"entity-resolution","task_name":"Entity Resolution"},{"task_slug":"feature-engineering","task_name":"Feature Engineering"},{"task_slug":"missing-values","task_name":"Missing Values"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/entity-resolution-on-abt-buy","task":"Entity Resolution","dataset":"Abt-Buy","model":"D-HAT","rank_in_archive_order":16,"of":16,"metrics":{"F1 (%)":"53.9"},"uses_additional_data":false},{"leaderboard":"/sota/entity-resolution-on-amazon-google","task":"Entity Resolution","dataset":"Amazon-Google","model":"D-HAT","rank_in_archive_order":10,"of":17,"metrics":{"F1 (%)":"67.5"},"uses_additional_data":false}],"syntology":{"atlas_url":null,"mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}