Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
HDFS >> mail # dev >> profiling hdfs write path


Copy link to this message
-
Re: profiling hdfs write path

> Currently it's CPU-intensive for several reasons:
> 1) It doesn't yet use the native CRC code
> 2) It makes several unnecessary copies and byte buffer allocations, both in
> the client and in the DataNode
>
> There are open JIRAs for these, and I have a preliminary patch which helped
> a lot, but it hasn't been high priority.
can you attach crc path there?
https://issues.apache.org/jira/browse/HDFS-3528
i will finish it.