{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/optimal-densification-for-fast-and-accurate","title":"Optimal Densification for Fast and Accurate Minwise Hashing","arxiv_id":"1703.04664","date":"2017-03-14","proceeding":"ICML 2017 8","authors":["Anshumali Shrivastava"],"abstract":"Minwise hashing is a fundamental and one of the most successful hashing\nalgorithm in the literature. Recent advances based on the idea of\ndensification~\\cite{Proc:OneHashLSH_ICML14,Proc:Shrivastava_UAI14} have shown\nthat it is possible to compute $k$ minwise hashes, of a vector with $d$\nnonzeros, in mere $(d + k)$ computations, a significant improvement over the\nclassical $O(dk)$. These advances have led to an algorithmic improvement in the\nquery complexity of traditional indexing algorithms based on minwise hashing.\nUnfortunately, the variance of the current densification techniques is\nunnecessarily high, which leads to significantly poor accuracy compared to\nvanilla minwise hashing, especially when the data is sparse. In this paper, we\nprovide a novel densification scheme which relies on carefully tailored\n2-universal hashes. We show that the proposed scheme is variance-optimal, and\nwithout losing the runtime efficiency, it is significantly more accurate than\nexisting densification techniques. As a result, we obtain a significantly\nefficient hashing scheme which has the same variance and collision probability\nas minwise hashing. Experimental evaluations on real sparse and\nhigh-dimensional datasets validate our claims. We believe that given the\nsignificant advantages, our method will replace minwise hashing implementations\nin practice.","url_abs":"http://arxiv.org/abs/1703.04664v1","url_pdf":"http://arxiv.org/pdf/1703.04664v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"optimal-densification-for-fast-and-accurate","repo_url":"https://github.com/chenghaomou/text-dedup","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":null}],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":null,"atlas_url":null,"mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}