Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
HDFS >> mail # dev >> murmur3 instead of crc32

Radim Kolar 2012-11-26, 02:08
Radim Kolar 2012-11-26, 02:58
Copy link to this message
Re: murmur3 instead of crc32
Hi Radim,

With SSE4.2 support, the iSCSI CRC32C is the fastest method available. As
of HDFS 2, we use that method by default for new files.


On Sun, Nov 25, 2012 at 6:58 PM, Radim Kolar <[EMAIL PROTECTED]> wrote:

> its not that big speed difference in this test:
> http://www.strchr.com/hash_**functions#results<http://www.strchr.com/hash_functions#results>
> asm version of CRC32 on i5 is fastest, but Java8 switched to murmur3 for
> hashing strings, i didnt get why they use it instead of
> *java.util.zip.CRC32. The collisions seems to be about same.*
Todd Lipcon
Software Engineer, Cloudera