Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
HBase >> mail # user >> HMaster not failing over dead RegionServers


Copy link to this message
-
Re: HMaster not failing over dead RegionServers
Thanks a bunch for the insight.  This message was actually coming from
master, but it still needs to grab the HLog files from hdfs, so I can still
see it being what you mentioned.  I'm going to look into tuning these
parameters down in preparation for future failures.

On Mon, Jul 2, 2012 at 7:56 PM, Suraj Varma <[EMAIL PROTECTED]> wrote:

> This looks like it is trying to reach a datanode ... doesn't it?
> > 12/06/30 00:07:22 INFO ipc.Client: Retrying connect to server: /
> 10.125.18.129:50020. Already tried 14 time(s).
>
> Is this from a master log or from a region server log? (I'm guess the
> above is from a region server log while trying to replay hlogs)
>
> Sometime back, we had a similar symptom (HLog splitting takes the long
> time due to the retries) and found that even though the datanode died,
> it was not being detected by the namenode. This leads to the region
> server retrying over dead datanodes over and over stretching out the
> splitting process.
>
> See this thread:
> http://www.mail-archive.com/[EMAIL PROTECTED]/msg10033.html
>
> We found that by default, it takes 15 mins for a datanode death to be
> detected by a NN ... and this seems to cause the NN serving back the
> dead DN as a valid one when RS tries to read the hlogs.
> The parameters in question are: dfs.heartbeat.recheck.interval and
> heartbeat.recheck.interval ... tweaking this down caused the recovery
> to be much faster.
> Also - hbase.rpc.timeout and zookeeper.session.timeout are two other
> configurations that need to be tweaked down from defaults for quick
> recovery.
>
> Not sure if this is the case in your error - but, might be something
> to investigate ...
> --Suraj
>
>
> On Sat, Jun 30, 2012 at 8:53 AM, Jimmy Xiang <[EMAIL PROTECTED]> wrote:
> > Bryan,
> >
> > The master could not detect if the region server is dead.
> > How do you set the zookeeper session timeout?
> >
> > Thanks,
> > Jimmy
> >
> > On Sat, Jun 30, 2012 at 8:09 AM, Stack <[EMAIL PROTECTED]> wrote:
> >> On Sat, Jun 30, 2012 at 7:04 AM, Bryan Beaudreault
> >> <[EMAIL PROTECTED]> wrote:
> >>> 12/06/30 00:07:22 INFO ipc.Client: Retrying connect to server: /
> >>> 10.125.18.129:50020. Already tried 14 time(s).
> >>>
> >>
> >> This was one of the servers that went down?
> >>
> >>> It was not following through the splitting of HLog files and didn't
> appear
> >>> to be moving regions off failed hosts.  After giving it about 20
> minutes to
> >>> try to right itself, I tried restarting the service.  The restart
> script
> >>> just hung for a while printing dots and nothing apparent was happening
> on
> >>> the logs at the time.
> >>
> >> Can we see the log  Bryan?
> >>
> >> You might thread dump when its hung-up the next time Bryan (Would be
> >> something for us to do a looksee on).
> >>
> >>> Finally I kill -9 the process, so that another
> >>> master could take over.  The new master seemed to start splitting
> logs, but
> >>> eventually got into the same state of printing the above message.
> >>>
> >>
> >> You think it a particular log?
> >>
> >>
> >>> Eventually it all worked out, but it took WAY too long (almost an
> hour, all
> >>> said).  Is this something that is tunable?
> >>
> >> Have RS carry less WALs?  Its a configuration.
> >>
> >>> They should have instantly been
> >>> removed from the list instead of retrying so many times.  Each server
> was
> >>> retried upwards of 30-40 times.
> >>>
> >>
> >> Yeah, thats a bit silly.
> >>
> >> We're working on the MTTR in general.  You logs would be of interest
> >> to a few of us if its ok that someone else can take a look.
> >>
> >> St.Ack
> >>
> >>> I am running cdh3u2 (0.90.4).
> >>>
> >>> Thanks,
> >>>
> >>> Bryan
>
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB