Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
HBase, mail # dev - HBase log grows very fast after stopped hadoop (due to connection exception)


Copy link to this message
-
Re: HBase log grows very fast after stopped hadoop (due to connection exception)
Stack 2014-02-05, 00:38
Logging at the rate you report is obnoxious.  We should recognize HDFS is
gone and backoff some.
St.Ack
On Tue, Feb 4, 2014 at 2:09 PM, Demai Ni <[EMAIL PROTECTED]> wrote:

> hi,
>
> we are using hbase 96.0(also saw the same issue on 94.x) on single node
> cluster. At some point, we stopped Hadoop, but keep hbase rerunning. As
> expected, hbase began to throw following errors:
> ..
> 2014-02-04 11:05:12,820 ERROR
> org.apache.hadoop.hbase.executor.EventHandler: Caught throwable while
> processing event M_META_SERVER_SHUTDOWN
> ......
> Caused by: java.net.ConnectException: Call From
> bdvm311.svl.ibm.com/9.30.123.64 to bdvm311.svl.ibm.com:9000 failed on
> connection exception: java.net.ConnectException: Connection refused; For
> more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
> ..
>
>  the whole exception log is pasted here: http://pastebin.com/9jfvfSmA
>
> while everything(error/exception) is valid, hbase keep retrying, and the
> log files grow to 25 GB within 3 hours.
>
> So, I am wondering whether there is a configuration to reduce the frequency
> of retry, or maybe we can twist the code to do it automatically?
>
> thank for your suggestion
>
> Demai
>