{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/fast-similarity-sketching","title":"Fast Similarity Sketching","arxiv_id":"1704.04370","date":"2017-04-14","proceeding":null,"authors":["Søren Dahlgaard","Mathias Bæk Tejs Langhede","Jakob Bæk Tejs Houen","Mikkel Thorup"],"abstract":"We consider the $\\textit{Similarity Sketching}$ problem: Given a universe $[u] = \\{0,\\ldots, u-1\\}$ we want a random function $S$ mapping subsets $A\\subseteq [u]$ into vectors $S(A)$ of size $t$, such that the Jaccard similarity $J(A,B) = |A\\cap B|/|A\\cup B|$ between sets $A$ and $B$ is preserved. More precisely, define $X_i = [S(A)[i] = S(B)[i]]$ and $X = \\sum_{i\\in [t]} X_i$. We want $E[X_i]=J(A,B)$, and we want $X$ to be strongly concentrated around $E[X] = t \\cdot J(A,B)$ (i.e. Chernoff-style bounds). This is a fundamental problem which has found numerous applications in data mining, large-scale classification, computer vision, similarity search, etc. via the classic MinHash algorithm. The vectors $S(A)$ are also called $\\textit{sketches}$. Strong concentration is critical, for often we want to sketch many sets $B_1,\\ldots,B_n$ so that we later, for a query set $A$, can find (one of) the most similar $B_i$. It is then critical that no $B_i$ looks much more similar to $A$ due to errors in the sketch. The seminal $t\\times\\textit{MinHash}$ algorithm uses $t$ random hash functions $h_1,\\ldots, h_t$, and stores $\\left ( \\min_{a\\in A} h_1(A),\\ldots, \\min_{a\\in A} h_t(A) \\right )$ as the sketch of $A$. The main drawback of MinHash is, however, its $O(t\\cdot |A|)$ running time, and finding a sketch with similar properties and faster running time has been the subject of several papers. (continued...)","url_abs":"https://arxiv.org/abs/1704.04370v4","url_pdf":"https://arxiv.org/pdf/1704.04370v4.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"links_only","authors_date_abstract":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license), from the Kaggle arXiv metadata snapshot of 2026-09-12"},"code_links":[{"paper_slug":"fast-similarity-sketching","repo_url":"https://github.com/chenghaomou/text-dedup","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":null}],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":null,"mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}