{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/stream-vbyte-faster-byte-oriented-integer","title":"Stream VByte: Faster Byte-Oriented Integer Compression","arxiv_id":"1709.08990","date":"2017-09-25","proceeding":null,"authors":["Daniel Lemire","Nathan Kurz","Christoph Rupp"],"abstract":"Arrays of integers are often compressed in search engines. Though there are\nmany ways to compress integers, we are interested in the popular byte-oriented\ninteger compression techniques (e.g., VByte or Google's Varint-GB). They are\nappealing due to their simplicity and engineering convenience. Amazon's\nvarint-G8IU is one of the fastest byte-oriented compression technique published\nso far. It makes judicious use of the powerful single-instruction-multiple-data\n(SIMD) instructions available in commodity processors. To surpass varint-G8IU,\nwe present Stream VByte, a novel byte-oriented compression technique that\nseparates the control stream from the encoded data. Like varint-G8IU, Stream\nVByte is well suited for SIMD instructions. We show that Stream VByte decoding\ncan be up to twice as fast as varint-G8IU decoding over real data sets. In this\nsense, Stream VByte establishes new speed records for byte-oriented integer\ncompression, at times exceeding the speed of the memcpy function. On a 3.4GHz\nHaswell processor, it decodes more than 4 billion differentially-coded integers\nper second from RAM to L1 cache.","url_abs":"http://arxiv.org/abs/1709.08990v2","url_pdf":"http://arxiv.org/pdf/1709.08990v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"stream-vbyte-faster-byte-oriented-integer","repo_url":"https://github.com/lemire/streamvbyte","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"none","reach":null},{"paper_slug":"stream-vbyte-faster-byte-oriented-integer","repo_url":"https://github.com/as-com/varint-simd","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":null},{"paper_slug":"stream-vbyte-faster-byte-oriented-integer","repo_url":"https://github.com/bmkessler/streamvbyte","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":null},{"paper_slug":"stream-vbyte-faster-byte-oriented-integer","repo_url":"https://github.com/lemire/MaskedVByte","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":null},{"paper_slug":"stream-vbyte-faster-byte-oriented-integer","repo_url":"https://github.com/nuclearfurnace/stream-vbyte","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":null},{"paper_slug":"stream-vbyte-faster-byte-oriented-integer","repo_url":"https://github.com/pombredanne/bitbucket.org-marshallpierce-stream-vbyte-rust","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":null}],"tasks":[],"methods":[{"method_slug":"speed","method_name":"SPEED"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":null,"mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}