Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
HDFS, mail # dev - murmur3 instead of crc32

Copy link to this message
Re: murmur3 instead of crc32
Todd Lipcon 2012-11-26, 03:38
Hi Radim,

With SSE4.2 support, the iSCSI CRC32C is the fastest method available. As
of HDFS 2, we use that method by default for new files.


On Sun, Nov 25, 2012 at 6:58 PM, Radim Kolar <[EMAIL PROTECTED]> wrote:

> its not that big speed difference in this test:
> http://www.strchr.com/hash_**functions#results<http://www.strchr.com/hash_functions#results>
> asm version of CRC32 on i5 is fastest, but Java8 switched to murmur3 for
> hashing strings, i didnt get why they use it instead of
> *java.util.zip.CRC32. The collisions seems to be about same.*
Todd Lipcon
Software Engineer, Cloudera