Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
HBase >> mail # dev >> HBase 0.94.15: writes stalls periodically even under moderate steady load (AWS EC2)


+
Vladimir Rodionov 2014-01-15, 20:33
+
Nick Dimiduk 2014-01-15, 22:14
+
Andrew Purtell 2014-01-15, 23:26
+
Andrew Purtell 2014-01-15, 23:27
+
Vladimir Rodionov 2014-01-16, 01:32
Copy link to this message
-
Re: HBase 0.94.15: writes stalls periodically even under moderate steady load (AWS EC2)
On Wed, Jan 15, 2014 at 5:32 PM,
Vladimir Rodionov <[EMAIL PROTECTED]> wrote:

> Yes, I am using ephemeral (local) storage. I found that iostat is most of
> the time idle on 3K load with periodic bursts up to 10% iowait.
>

Ok, sounds like the problem is higher up the stack.

I see in later emails on this thread a log snippet that shows an issue with
the WAL writer pipeline, one of the datanodes is slow, sick, or partially
unreachable. If you have uneven point to point ping times among your
cluster instances, or periodic loss, it might still be AWS's fault,
otherwise I wonder why the DFSClient says a datanode is sick.

--
Best regards,

   - Andy

Problems worthy of attack prove their worth by hitting back. - Piet Hein
(via Tom White)
+
Bryan Beaudreault 2014-01-16, 18:33
+
lars hofhansl 2014-01-16, 20:58
+
Vladimir Rodionov 2014-01-16, 21:08
+
Michael Segel 2014-01-17, 11:55
+
Michael Segel 2014-01-17, 15:03
+
Nick Dimiduk 2014-01-17, 18:54
+
Ted Yu 2014-01-17, 19:11
+
Andrew Purtell 2014-01-17, 18:21
+
Vladimir Rodionov 2014-01-17, 18:33
+
lars hofhansl 2014-01-16, 05:17
+
Vladimir Rodionov 2014-01-16, 05:49
+
谢良 2014-01-16, 05:55
+
Vladimir Rodionov 2014-01-16, 06:45
+
lars hofhansl 2014-01-16, 07:13
+
谢良 2014-01-16, 08:24
+
Vladimir Rodionov 2014-01-16, 14:46
+
lars hofhansl 2014-01-17, 16:49
+
Vladimir Rodionov 2014-01-15, 22:07
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB