Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
HDFS >> mail # dev >> murmur3 instead of crc32


Copy link to this message
-
Re: murmur3 instead of crc32
Hi Radim,

With SSE4.2 support, the iSCSI CRC32C is the fastest method available. As
of HDFS 2, we use that method by default for new files.

-Todd

On Sun, Nov 25, 2012 at 6:58 PM, Radim Kolar <[EMAIL PROTECTED]> wrote:

> its not that big speed difference in this test:
>
> http://www.strchr.com/hash_**functions#results<http://www.strchr.com/hash_functions#results>
>
> asm version of CRC32 on i5 is fastest, but Java8 switched to murmur3 for
> hashing strings, i didnt get why they use it instead of
>
> *java.util.zip.CRC32. The collisions seems to be about same.*
>
>
--
Todd Lipcon
Software Engineer, Cloudera