Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
HDFS >> mail # dev >> profiling hdfs write path


+
Radim Kolar 2012-11-25, 21:41
Copy link to this message
-
Re: profiling hdfs write path
Hi Radim,

Currently it's CPU-intensive for several reasons:
1) It doesn't yet use the native CRC code
2) It makes several unnecessary copies and byte buffer allocations, both in
the client and in the DataNode

There are open JIRAs for these, and I have a preliminary patch which helped
a lot, but it hasn't been high priority. On most clusters, writing becomes
network bound before being CPU-bound. On the other hand, as 10gbe is
becoming fairly common, this will probably be more important soon. Hoping
to find time to get back to finishing the patches in the next few months.

-Todd

On Sun, Nov 25, 2012 at 1:41 PM, Radim Kolar <[EMAIL PROTECTED]> wrote:

> anybody tried to profile why HDFS write path is so much CPU intensive?
>

--
Todd Lipcon
Software Engineer, Cloudera
+
Radim Kolar 2012-11-29, 15:17
+
Todd Lipcon 2012-11-29, 18:25
+
Radim Kolar 2012-12-04, 17:07
+
Todd Lipcon 2012-12-04, 17:27
+
Suresh Srinivas 2012-12-04, 17:49
+
Radim Kolar 2012-12-04, 17:39
+
Eli Collins 2012-12-04, 17:44
+
Radim Kolar 2012-12-05, 02:00
+
Andy Isaacson 2012-12-05, 22:21
+
Radim Kolar 2012-12-06, 02:02
+
Andy Isaacson 2012-12-06, 23:06
+
Radim Kolar 2012-12-08, 04:39
+
Steve Loughran 2012-12-08, 12:38
+
Steve Loughran 2012-12-05, 08:57
+
Radim Kolar 2012-11-26, 03:07
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB