Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
HBase >> mail # user >> how to make tuning for hbase (every couple of days hbase region sever/s crashe)


Copy link to this message
-
Re: how to make tuning for hbase (every couple of days hbase region sever/s crashe)
Are you doing some intensive tasks at the RegionServer side (which takes
more than default time out of, 60 sec -> iirc)?
One can get these exception when client side socket connection is closed
(probably a time out from client side).
As per the exception, when RegionServer tried to send the result of a Multi
call via its handler thread, the other side of the socket was closed and it
failed to write any response.

Cheers,
Himanshu

On Tue, Aug 23, 2011 at 5:35 PM, Buttler, David <[EMAIL PROTECTED]> wrote:

> So, if you use 0.5 GB / mapper and 1 GB / reducer, your total memory
> consumption (minus hbase) on a slave node should be:
> 4 GB M/R tasks
> 1 GB OS -- just a guess
> 1 GB datanode
> 1 GB tasktracker
> Leaving you with up to 9 GB for your region servers.  I would suggest
> bumping your region server ram up to 8GB, and leave a GB for OS caching. [I
> am sure someone out there will tell me I am crazy]
>
>
> However, it is the log that is the most useful part of your email.
>  Unfortunately I haven't seen that error before.
> Are you using the Multi methods a lot in your code?
>
> Dave
>
> -----Original Message-----
> From: Oleg Ruchovets [mailto:[EMAIL PROTECTED]]
> Sent: Tuesday, August 23, 2011 1:38 PM
> To: [EMAIL PROTECTED]
> Subject: Re: how to make tuning for hbase (every couple of days hbase
> region sever/s crashe)
>
> Thank you for detailed response,
>
> On Tue, Aug 23, 2011 at 7:49 PM, Buttler, David <[EMAIL PROTECTED]> wrote:
>
> > Have you looked at the logs of the region servers?  That is a good first
> > place to look.
>
> How many regions are in your system?
>
>
>         Region Servers
>
> Address Start Code Load
> hadoop01 1314007529600 requests=0, regions=212, usedHeap=3171, maxHeap=3983
> hadoop02 1314007496109 requests=0, regions=207, usedHeap=2185, maxHeap=3983
> hadoop03 1314008874001 requests=0, regions=208, usedHeap=1955, maxHeap=3983
> hadoop04 1314008965432 requests=0, regions=209, usedHeap=2034, maxHeap=3983
> hadoop05 1314007496533 requests=0, regions=208, usedHeap=1970, maxHeap=3983
> hadoop06 1314008874036 requests=0, regions=208, usedHeap=1987, maxHeap=3983
> hadoop07 1314007496927 requests=0, regions=209, usedHeap=2118, maxHeap=3983
> hadoop08 1314007497034 requests=0, regions=211, usedHeap=2568, maxHeap=3983
> hadoop09 1314007497221 requests=0, regions=209, usedHeap=2148, maxHeap=3983
> master            1314008873765 requests=0, regions=208, usedHeap=2007,
> maxHeap=3962
> Total: servers: 10  requests=0, regions=2089
>
> most of the  time GC succeeded to clean up but every 3/4 days used memory
> become close to 4G
>
> and there are alot of Exceptions like this:
>
>   org.apache.hadoop.ipc.*HBase*Server: IPC Server
> Responder, call multi(org.apache.hadoop.*hbase*.client.MultiAction@491fb2f4
> )
> from 10.11.87.73:33737: output error
> 2011-08-14 18:37:36,264 WARN org.apache.hadoop.ipc.*HBase*Server: IPC
> Server
> handler 24 on 8041 caught: java.nio.channels.ClosedChannelException
>        at
> sun.nio.ch.SocketChannelImpl.ensureWriteOpen(SocketChannelImpl.java:133)
>        at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:324)
>        at
> org.apache.hadoop.*hbase.ipc.HBaseServer.channelIO(HBase*Server.java:1387)
>        at
> org.apache.hadoop.*hbase.ipc.HBaseServer.channelWrite(HBase*
> Server.java:1339)
>        at
> org.apache.hadoop.*hbase.ipc.HBaseServer$Responder.processResponse(HBase*
> Server.java:727)
>        at
> org.apache.hadoop.*hbase.ipc.HBaseServer$Responder.doRespond(HBase*
> Server.java:792)
>        at
>
> org.apache.hadoop.*hbase.ipc.HBaseServer$Handler.run(HBase*Server.java:1083)
>
>
>
>
>
>
> >  If you are using MSLAB, it reserves 2MB/region as a buffer -- that can
> add
> > up when you have lots of regions.
> >
> >
>
>
>
> > Given so little information all my guesses are going to be wild, but they
> > might help:
> > 4GB may not be enough for your current load.
>
> Have you considered changing your memory allocation, giving less to your
> > map/reduce jobs and more to HBase?
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB