Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
HDFS >> mail # dev >> profiling hdfs write path


Copy link to this message
-
Re: profiling hdfs write path

> Currently it's CPU-intensive for several reasons:
> 1) It doesn't yet use the native CRC code
> 2) It makes several unnecessary copies and byte buffer allocations, both in
> the client and in the DataNode
>
> There are open JIRAs for these, and I have a preliminary patch which helped
> a lot, but it hasn't been high priority.
can you attach crc path there?
https://issues.apache.org/jira/browse/HDFS-3528
i will finish it.
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB