Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hadoop, mail # general - Add Append-HBase support in upcoming 20.205


Copy link to this message
-
Re: Add Append-HBase support in upcoming 20.205
Andrew Purtell 2011-09-02, 05:56
> From: Arun C Murthy <[EMAIL PROTECTED]>

> +1
>
> I think it's high time (I thought so too last Dec too:
> http://s.apache.org/jr) we had an official Hadoop release which supports
> HBase. Thanks for all the effort Andrew - it will be really nice to have
> an Apache 0.20 with security+append!

Thanks to Dhruba, Todd, Hairong, and the other original contributors of append support to HDFS 0.20.x.

I'm +1 obviously. :-)
Best regards,
    - Andy

Problems worthy of attack prove their worth by hitting back. - Piet Hein (via Tom White)
----- Original Message -----
> From: Arun C Murthy <[EMAIL PROTECTED]>
> To: [EMAIL PROTECTED]
> Cc:
> Sent: Friday, September 2, 2011 6:36 AM
> Subject: Re: Add Append-HBase support in upcoming 20.205
>
>
> On Aug 31, 2011, at 11:41 AM, sanjay Radia wrote:
>
>>
>>  I propose that the 20-append patches (details below)  be included in 20.205
> which will become the first official Apache
>>  release of Hadoop that supports Append and HBase.
>>
>
>>  Background:
>>  There hasn't been a official Apache release that supports HBase.
>>  The HBase community have instead been using the 20-append branch; the
> patches were contributed by the HBase community including Facebook. The Cloudera
> distribution has also included these patches.
>>  Andrew Purtell has ported these patches to 20-security branch.
>>
>
>
> +1
>
> I think it's high time (I thought so too last Dec too:
> http://s.apache.org/jr) we had an official Hadoop release which supports HBase.
> Thanks for all the effort Andrew - it will be really nice to have an Apache 0.20
> with security+append!
>
> Arun
>
>>  Risk Level:
>>  These patches have been used and tested on large HBase clusters by FB , by
> those who use 20-append branch directly (various users including a 500 node
> HBase cluster at Yahoo) and by those that use the Cloudera distribution. We have
> reviewed the patches and have conducted further tests; testing and validation
> continues.
>>
>>
>>  Patches:
>>  HDFS-200. Support append and sync for hadoop 0.20 branch.
>>  HDFS-142. Blocks that are being written by a client are stored in the
> blocksBeingWritten directory.
>>  HDFS-1057.  Concurrent readers hit ChecksumExceptions if following a writer
> to very end of file
>>  HDFS-724.  Use a bidirectional heartbeat to detect stuck pipeline.
>>  HDFS-895. Allow hflush/sync to occur in parallel with new writes to the
> file.
>>  HDFS-1520. Lightweight NameNode operation recoverLease to trigger lease
> recovery.
>>  HDFS-1555. Disallow pipelien recovery if a file is already being lease
> recovered.
>>  HDFS-1554. New semantics for recoverLease.
>>  HDFS-988. Fix bug where savenameSpace can corrupt edits log.
>>  HDFS-826. Allow a mechanism for an application to detect that datanode(s)
> have died in the write pipeline.
>>  HDFS-630. Client can exclude specific nodes in the write pipeline.
>>  HDFS-1141. completeFile does not check lease ownership.
>>  HDFS-1204. Lease expiration should recover single files, not entire lease
> holder
>>  HDFS-1254. Support append/sync via the default configuration.
>>  HDFS-1346. DFSClient receives out of order packet ack.
>>  HDFS-1054. remove sleep before retry for allocating a block.
>>
>