{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/recursive-n-gram-hashing-is-pairwise","title":"Recursive n-gram hashing is pairwise independent, at best","arxiv_id":"0705.4676","date":"2007-05-31","proceeding":null,"authors":["Daniel Lemire","Owen Kaser"],"abstract":"Many applications use sequences of n consecutive symbols (n-grams). Hashing\nthese n-grams can be a performance bottleneck. For more speed, recursive hash\nfamilies compute hash values by updating previous values. We prove that\nrecursive hash families cannot be more than pairwise independent. While hashing\nby irreducible polynomials is pairwise independent, our implementations either\nrun in time O(n) or use an exponential amount of memory. As a more scalable\nalternative, we make hashing by cyclic polynomials pairwise independent by\nignoring n-1 bits. Experimentally, we show that hashing by cyclic polynomials\nis is twice as fast as hashing by irreducible polynomials. We also show that\nrandomized Karp-Rabin hash families are not pairwise independent.","url_abs":"http://arxiv.org/abs/0705.4676v8","url_pdf":"http://arxiv.org/pdf/0705.4676v8.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"recursive-n-gram-hashing-is-pairwise","repo_url":"https://github.com/lemire/rollinghashcpp","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":null},{"paper_slug":"recursive-n-gram-hashing-is-pairwise","repo_url":"https://github.com/zhaoxiaofei/bindash","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":null}],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":null,"atlas_url":null,"mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}