Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
HDFS >> mail # dev >> FW: Handling read failures during recovery


Copy link to this message
-
Re: FW: Handling read failures during recovery
>> If read request comes before completing client recovery process, do we
need to make the read operation wait until recovery completes successfully?
That would be desirable.

On Tue, Aug 9, 2011 at 7:08 AM, Uma Maheswara Rao G 72686 <
[EMAIL PROTECTED]> wrote:

>
>
> Hi All,
>
> Any thoughts?
>
>  Looks Hbase is going to address this issue.
> https://issues.apache.org/jira/browse/HBASE-4177.
>
> Do we need to address from HDFS as well?
>
> If read request comes before completing client recovery process, do we need
> to make the read operation wait until recovery completes successfully?
>
>
> Regards,
> Uma
> > -----Original Message-----
> > From: Ramkrishna S Vasudevan [mailto:[EMAIL PROTECTED]]
> > Sent: Friday, August 05, 2011 9:52 AM
> > To: [EMAIL PROTECTED]; [EMAIL PROTECTED]
> > Subject: RE: Handling read failures during recovery
> >
> > Hi
> >
> > As Laxman pointed out, there is a potential problem here.  We
> > expect the
> > Namenode recovery to happen within a specified time and we tend to
> > sleep for
> > one second in the splitLogs logic.  But we carry on with reading
> > the HLog
> > file which will result in failure.  So if the logs are not split
> > properlythere could be a data loss.
> >
> >
> >
> > Regards
> > Ram
> >
> >
> >
> > -----Original Message-----
> > From: Laxman [mailto:[EMAIL PROTECTED]]
> > Sent: Tuesday, August 02, 2011 10:47 AM
> > To: [EMAIL PROTECTED]; [EMAIL PROTECTED]
> > Subject: FW: Handling read failures during recovery
> >
> > Partial mail was sent accidentally. Sorry for that.
> > Resending with complete details, analysis and logs.
> >
> > 20-append version we are using.
> >
> > To summarize there are two problems [One each from HDFS and HBase] we
> > noticed in this flow.
> >
> >
> > 1) From HDFS
> > Even though client is getting the updated block info from Namenode
> > on first
> > read failure, client is discarding the new info and using the old
> > info only
> > to retrieve the data from datanode. So, all the read
> > retries are failing. [Method parameter reassignment - Not
> > reflected in
> > caller]
> >
> >
> > HDFS Code snippet
> > org.apache.hadoop.hdfs.DFSClient.DFSInputStream.chooseDataNode
> >
> > private DNAddrPair chooseDataNode(LocatedBlock block)
> >      throws IOException {
> > ...
> > ...
> > block = getBlockAt(block.getStartOffset(), false);
> > ...
> > ...
> > }
> >
> > Here method parameter "block" is assigned with the new block info
> > which is
> > not reflected in the caller "blockSeekTo(long target)".
> >
> > 2) From HBase
> >
> > Excerpt from my previous mail.
> >
> > > As the recovery is an asynchronous operation recoverLease call
> > will return
> > > immediately and may end up with read failure as the recovery is in
> > progress.
> > >
> > > This may lead to some regions to be in offline state only
> >
> > > One approach is to introduce a delay in between recovery and
> > read. But,
> > this
> > > may not be a fool proof way to address this.
> >
> > I've noticed the delay is already present in HBase code. But as I
> > mentionedthis may not be a fool proof mechanism to handle this
> > scenario.
> > HBase Code snippet
> > In the class HLogSplitter the splitLog() calls recoverFileLease().
> >
> > In recoverFileLease()
> >
> >      try {
> >        Thread.sleep(1000);
> >      } catch (InterruptedException ex) {
> >        new InterruptedIOException().initCause(ex);
> >      }
> >
> > Once the recover call is made we sleep for one sec and proceed with
> > parseHLog().
> >
> >
> > Here is the log
> > 2011-07-21 17:01:19,642 INFO org.apache.hadoop.hdfs.DFSClient:
> > Could not
> > obtain block blk_1311262402613_3094 from any node:
> > java.io.IOException: No
> > live nodes contain current block. Will get new block locations
> > from namenode
> > and retry...
> > 2011-07-21 17:01:20,650 INFO org.apache.hadoop.hdfs.DFSClient:
> > Could not
> > obtain block blk_1311262402613_3094 from any node:
> > java.io.IOException: No
> > live nodes contain current block. Will get new block locations
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB