Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
HBase >> mail # user >> Region servers going down under heavy write load


Copy link to this message
-
Re: Region servers going down under heavy write load
bq.  I just dont find this "hbase.zookeeper.property.tickTime" anywhere in
the code base.

Neither do I. Mind filing a JIRA to correct this in troubleshooting.xml ?

bq.  increase tickTime in zoo.cfg?

For shared zookeeper quorum, the above should be done.

On Wed, Jun 5, 2013 at 5:45 PM, Ameya Kantikar <[EMAIL PROTECTED]> wrote:

> One more thing. I just dont find this "hbase.zookeeper.property.tickTime"
> anywhere in the code base.
> Also, I could not find ZooKeeper API that takes tickTime from client.
>
> http://zookeeper.apache.org/doc/r3.3.3/api/org/apache/zookeeper/ZooKeeper.html
> It takes sessionTime out value, but not tickTime.
>
> Is this even relevant anymore? hbase.zookeeper.property.tickTime ?
>
> So whats the solution, increase tickTime in zoo.cfg? (and not
> hbase.zookeeper.property.tickTime
> in hbase-site.xml?)
>
> Ameya
>
>
> On Wed, Jun 5, 2013 at 3:18 PM, Ameya Kantikar <[EMAIL PROTECTED]> wrote:
>
> > Which tickTime is honored?
> >
> > One in zoo.cfg or hbase.zookeeper.property.tickTime in hbase-site.xml?
> >
> > My understanding now is, whichever tickTime is honored, session time can
> > not be more than 20 times the value.
> >
> > I think this is whats happening on my cluster:
> >
> > My hbase.zookeeper.property.tickTime value is 6000 ms. However my timeout
> > value is 300000 ms which is outside of 20 times tickTime. Hence ZooKeeper
> > uses its syncLimit of 5, to generate 6000*5 = 30000 as timeout value for
> my
> > RS sessions.
> >
> > I'll try increasing hbase.zookeeper.property.tickTime value in
> > hbase-site.xml and will monitor my cluster over next few days.
> >
> > Thanks Kevin & Ted for your help.
> >
> > Ameya
> >
> >
> >
> >
> > On Wed, Jun 5, 2013 at 2:45 PM, Ted Yu <[EMAIL PROTECTED]> wrote:
> >
> >> bq. I thought this property in hbase-site.xml takes care of that:
> >> zookeeper.session.timeout
> >>
> >> From
> >>
> >>
> http://zookeeper.apache.org/doc/current/zookeeperProgrammers.html#ch_zkSessions
> >> :
> >>
> >> The client sends a requested timeout, the server responds with the
> timeout
> >> that it can give the client. The current implementation requires that
> the
> >> timeout be a minimum of 2 times the tickTime (as set in the server
> >> configuration) and a maximum of 20 times the tickTime. The ZooKeeper
> >> client
> >> API allows access to the negotiated timeout.
> >> The above means the shared zookeeper quorum may return timeout value
> >> different from that of zookeeper.session.timeout
> >>
> >> Cheers
> >>
> >> On Wed, Jun 5, 2013 at 2:34 PM, Ameya Kantikar <[EMAIL PROTECTED]>
> wrote:
> >>
> >> > In zoo.cfg I have not setup this value explicitly. My zoo.cfg looks
> >> like:
> >> >
> >> > tickTime=2000
> >> > initLimit=10
> >> > syncLimit=5
> >> >
> >> > We use common zoo keeper cluster for 2 of our HBase clusters. I'll try
> >> > increasing this value from zoo.cfg.
> >> > However is it possible to set this value cluster specific?
> >> > I thought this property in hbase-site.xml takes care of that:
> >> > zookeeper.session.timeout
> >> >
> >> >
> >> > On Wed, Jun 5, 2013 at 1:49 PM, Kevin O'dell <
> [EMAIL PROTECTED]
> >> > >wrote:
> >> >
> >> > > Ameya,
> >> > >
> >> > >   What does your zoo.cfg say for your timeout value?
> >> > >
> >> > >
> >> > > On Wed, Jun 5, 2013 at 4:47 PM, Ameya Kantikar <[EMAIL PROTECTED]>
> >> > wrote:
> >> > >
> >> > > > Hi,
> >> > > >
> >> > > > We have heavy map reduce write jobs running against our cluster.
> >> Every
> >> > > once
> >> > > > in a while, we see a region server going down.
> >> > > >
> >> > > > We are on : 0.94.2-cdh4.2.0, r
> >> > > >
> >> > > > We have done some tuning for heavy map reduce jobs, and have
> >> increased
> >> > > > scanner timeouts, lease timeouts, have also tuned memstore as
> >> follows:
> >> > > >
> >> > > > hbase.hregion.memstore.block.multiplier: 4
> >> > > > hbase.hregion.memstore.flush.size: 134217728
> >> > > > hbase.hstore.blockingStoreFiles: 100
> >> > > >
> >> > > > So now, we are still facing issues. Looking at the logs it looks
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB