Hi All,

            I am running the HBase in fully distributed mode. I used the HBase 0.90.2 version . I have one doubt , that one is what will happen when the Region server is crashed ?
If the one region is managed by one region server then what will happen after the crash of the region server?

Thanks in Advance....
Regards
R.Shanmuganathan  

The master will detect that RS is down by periodically checking a
zookeeper ( it will say in the master log, znode expired ).  After, it
will check to see if there is anything in /hbase/.logs directory for
that region server, if something is found, master will replay the log
records and 'push' them into region directories, when master is done,
the 'unopened' regions will be opened by the rest of the region
servers and all regions that were down will be back online.

-Jack

On Wed, Sep 21, 2011 at 12:21 AM, shanmuganathan.r
<[EMAIL PROTECTED]> wrote:
> Hi All,
>
>            I am running the HBase in fully distributed mode. I used the HBase 0.90.2 version . I have one doubt , that one is what will happen when the Region server is crashed ?
>
>
> If the one region is managed by one region server then what will happen after the crash of the region server?
>
> Thanks in Advance....
>
>
> Regards
>
>
> R.Shanmuganathan
>
>
>

Hi Jack,

            I understood this concept. The DFS client is directly connected to the datanodes in hdfs and perform the read/write block data with communicate with the namenode by client protocol . Is there is no direct involvement of
 namenode in the HBase operation ?

Thanks

R.Shanmuganathan  

---- On Wed, 21 Sep 2011 12:55:37 +0530 Jack Levin<[EMAIL PROTECTED]> wrote ----
The master will detect that RS is down by periodically checking a
zookeeper ( it will say in the master log, znode expired ). After, it
will check to see if there is anything in /hbase/.logs directory for
that region server, if something is found, master will replay the log
records and 'push' them into region directories, when master is done,
the 'unopened' regions will be opened by the rest of the region
servers and all regions that were down will be back online.
 
-Jack
 
On Wed, Sep 21, 2011 at 12:21 AM, shanmuganathan.r
<[EMAIL PROTECTED]> wrote:
> Hi All,
>
>            I am running the HBase in fully distributed mode. I used the HBase 0.90.2 version . I have one doubt , that one is what will happen when the Region server is crashed ?
>
>
> If the one region is managed by one region server then what will happen after the crash of the region server?
>
> Thanks in Advance....
>
>
> Regards
>
>
> R.Shanmuganathan
>
>
>

That question is sort of orthogonal to the first one... Please create
a new thread if you want to change the subject.

HBase uses the DFSClient, for example if HBase needs to open a file it
needs to talk to the NN first (through the DFSClient, HBase doesn't
really know what's going on).

J-D

On Wed, Sep 21, 2011 at 2:06 AM, shanmuganathan.r
<[EMAIL PROTECTED]> wrote:
> Hi Jack,
>
>            I understood this concept. The DFS client is directly connected to the datanodes in hdfs and perform the read/write block data with communicate with the namenode by client protocol . Is there is no direct involvement of
>
>
>  namenode in the HBase operation ?
>
> Thanks
>
> R.Shanmuganathan
>
>
>
> ---- On Wed, 21 Sep 2011 12:55:37 +0530 Jack Levin<[EMAIL PROTECTED]> wrote ----
>
>
> The master will detect that RS is down by periodically checking a
> zookeeper ( it will say in the master log, znode expired ). After, it
> will check to see if there is anything in /hbase/.logs directory for
> that region server, if something is found, master will replay the log
> records and 'push' them into region directories, when master is done,
> the 'unopened' regions will be opened by the rest of the region
> servers and all regions that were down will be back online.
>
> -Jack
>
> On Wed, Sep 21, 2011 at 12:21 AM, shanmuganathan.r
> <[EMAIL PROTECTED]> wrote:
> > Hi All,
> >
> >            I am running the HBase in fully distributed mode. I used the HBase 0.90.2 version . I have one doubt , that one is what will happen when the Region server is crashed ?
> >
> >
> > If the one region is managed by one region server then what will happen after the crash of the region server?
> >
> > Thanks in Advance....
> >
> >
> > Regards
> >
> >
> > R.Shanmuganathan
> >
> >
> >
>
>

Hi,

I have three region servers running on datanodes, one of the region server
crashes when try to insert with below error and the other two region
servers are running without any errors
WARN org.apache.hadoop.hdfs.DFSClient: Error Recovery for block
blk_-2411272549088965456_2503 bad datanode[0] 10.177.78.15:50010
2011-11-28 11:49:29,951 WARN org.apache.hadoop.hdfs.DFSClient: Error while
syncing
java.io.IOException: All datanodes 10.177.78.15:50010 are bad. Aborting...
at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.processDatanodeError(DFSClient.java:2766)
at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$1600(DFSClient.java:2305)
at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2477)
2011-11-28 11:49:29,953 FATAL
org.apache.hadoop.hbase.regionserver.wal.HLog: Could not append. Requesting
close of hlog
java.io.IOException: Reflection
at
org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogWriter.sync(SequenceFileLogWriter.java:147)
at org.apache.hadoop.hbase.regionserver.wal.HLog.sync(HLog.java:986)
at
org.apache.hadoop.hbase.regionserver.wal.HLog$LogSyncer.run(HLog.java:963)
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogWriter.sync(SequenceFileLogWriter.java:145)
... 2 more
Caused by: java.io.IOException: All datanodes 10.177.78.15:50010 are bad.
Aborting...
at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.processDatanodeError(DFSClient.java:2766)
at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$1600(DFSClient.java:2305)
at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2477)
2011-11-28 11:49:29,954 ERROR
org.apache.hadoop.hbase.regionserver.wal.HLog: Error while syncing,
requesting close of hlog
java.io.IOException: Reflection
at
org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogWriter.sync(SequenceFileLogWriter.java:147)
at org.apache.hadoop.hbase.regionserver.wal.HLog.sync(HLog.java:986)
at
org.apache.hadoop.hbase.regionserver.wal.HLog$LogSyncer.run(HLog.java:963)
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogWriter.sync(SequenceFileLogWriter.java:145)
... 2 more
Caused by: java.io.IOException: All datanodes 10.177.78.15:50010 are bad.
Aborting...
at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.processDatanodeError(DFSClient.java:2766)
at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$1600(DFSClient.java:2305)
at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2477)
2011-11-28 11:49:29,954 INFO org.apache.hadoop.hbase.regionserver.wal.HLog:
regionserver60020.logSyncer exiting
2011-11-28 11:50:37,573 WARN org.apache.hadoop.hdfs.DFSClient: Error while
syncing
java.io.IOException: All datanodes 10.177.78.15:50010 are bad. Aborting...
at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.processDatanodeError(DFSClient.java:2766)
at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$1600(DFSClient.java:2305)
at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2477)
2011-11-28 11:50:37,574 FATAL
org.apache.hadoop.hbase.regionserver.wal.HLog: Could not append. Requesting
close of hlog

Thanks
Arun