Hi all,

I have recently seen silent data loss in our system.
Here is the case:

1. client appends to some block
2. for some reason, commitBlockSynchronization
    returns successfully with synclist = [] (i.e empty)
3. in the client code, NO exception is thrown, and
   client appends successfully.
4. However, the block replicas are then removed from
   datanodes, causing data loss.

Have any one seen this before?
Is this behavior by design or a bug?

Many thanks,
Thanh

What version are you using?

On Thu, Apr 14, 2011 at 3:55 PM, Thanh Do <[EMAIL PROTECTED]> wrote:

> Hi all,
>
> I have recently seen silent data loss in our system.
> Here is the case:
>
> 1. client appends to some block
> 2. for some reason, commitBlockSynchronization
>     returns successfully with synclist = [] (i.e empty)
> 3. in the client code, NO exception is thrown, and
>    client appends successfully.
> 4. However, the block replicas are then removed from
>    datanodes, causing data loss.
>
> Have any one seen this before?
> Is this behavior by design or a bug?
>
> Many thanks,
> Thanh
>
>
>
>

I am using cloudera's distribution version:
hadoop-0.20.2+738.

On Thu, Apr 14, 2011 at 6:23 PM, Ted Dunning <[EMAIL PROTECTED]> wrote:

> What version are you using?
>
>
> On Thu, Apr 14, 2011 at 3:55 PM, Thanh Do <[EMAIL PROTECTED]> wrote:
>
>> Hi all,
>>
>> I have recently seen silent data loss in our system.
>> Here is the case:
>>
>> 1. client appends to some block
>> 2. for some reason, commitBlockSynchronization
>>     returns successfully with synclist = [] (i.e empty)
>> 3. in the client code, NO exception is thrown, and
>>    client appends successfully.
>> 4. However, the block replicas are then removed from
>>    datanodes, causing data loss.
>>
>> Have any one seen this before?
>> Is this behavior by design or a bug?
>>
>> Many thanks,
>> Thanh
>>
>>
>>
>>
>

Hi Thanh,

There were some known bugs in the append feature in Apache Hadoop 0.19.  These
bugs were fixed in both Apache 0.20-append and Apache 0.21.  For Cloudera's
distribution, I have no idea.  You may want to ask your questions in Cloudera's
mailing lists.

I am sorry for the bugs.

Regards,
Nicholas

________________________________
From: Thanh Do <[EMAIL PROTECTED]>
To: Ted Dunning <[EMAIL PROTECTED]>
Cc: [EMAIL PROTECTED]; [EMAIL PROTECTED]
Sent: Fri, April 15, 2011 8:02:05 AM
Subject: Re: silent data loss during append

I am using cloudera's distribution version:
hadoop-0.20.2+738.

On Thu, Apr 14, 2011 at 6:23 PM, Ted Dunning <[EMAIL PROTECTED]> wrote:

> What version are you using?
>
>
> On Thu, Apr 14, 2011 at 3:55 PM, Thanh Do <[EMAIL PROTECTED]> wrote:
>
>> Hi all,
>>
>> I have recently seen silent data loss in our system.
>> Here is the case:
>>
>> 1. client appends to some block
>> 2. for some reason, commitBlockSynchronization
>>     returns successfully with synclist = [] (i.e empty)
>> 3. in the client code, NO exception is thrown, and
>>    client appends successfully.
>> 4. However, the block replicas are then removed from
>>    datanodes, causing data loss.
>>
>> Have any one seen this before?
>> Is this behavior by design or a bug?
>>
>> Many thanks,
>> Thanh