Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
HDFS >> mail # dev >> murmur3 instead of crc32


+
Radim Kolar 2012-11-26, 02:08
+
Radim Kolar 2012-11-26, 02:58
Copy link to this message
-
Re: murmur3 instead of crc32
Hi Radim,

With SSE4.2 support, the iSCSI CRC32C is the fastest method available. As
of HDFS 2, we use that method by default for new files.

-Todd

On Sun, Nov 25, 2012 at 6:58 PM, Radim Kolar <[EMAIL PROTECTED]> wrote:

> its not that big speed difference in this test:
>
> http://www.strchr.com/hash_**functions#results<http://www.strchr.com/hash_functions#results>
>
> asm version of CRC32 on i5 is fastest, but Java8 switched to murmur3 for
> hashing strings, i didnt get why they use it instead of
>
> *java.util.zip.CRC32. The collisions seems to be about same.*
>
>
--
Todd Lipcon
Software Engineer, Cloudera
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB