Hi :

Do data nodes, when they receive write requests, forward the request to the
next datanode in the chain after appending to a file? Or rather, do the
write and forward asynchronously and at the same time?

--
Jay Vyas
http://jayunit100.blogspot.com

As per my understanding it's sequential. Let's wait for some experts
comments.

Best Regards,
Tariq
+91-9741563634

On Wed, Dec 19, 2012 at 2:26 AM, Jay Vyas <[EMAIL PROTECTED]> wrote:

> Hi :
>
> Do data nodes, when they receive write requests, forward the request to
> the next datanode in the chain after appending to a file? Or rather, do the
> write and forward asynchronously and at the same time?
>
> --
> Jay Vyas
> http://jayunit100.blogspot.com
>

Hi,

The received write packet is directly socket-written to the next
node's receiver (async process - we don't wait for it to hit the disk
there), and then written onto the disk of self().

You may see HDFS classes BlockReceiver and PacketReceiver in trunk for
more information.

On Wed, Dec 19, 2012 at 2:26 AM, Jay Vyas <[EMAIL PROTECTED]> wrote:
> Hi :
>
> Do data nodes, when they receive write requests, forward the request to the
> next datanode in the chain after appending to a file? Or rather, do the
> write and forward asynchronously and at the same time?
>
> --
> Jay Vyas
> http://jayunit100.blogspot.com

--
Harsh J

Harsh,

Is that a change from 1.0 code?  Hortonworks explains it a little
differently.

Thanks for the details and pointer to the code.

Chris (another one)
On Dec 18, 2012 5:14 PM, "Harsh J" <[EMAIL PROTECTED]> wrote:

> Hi,
>
> The received write packet is directly socket-written to the next
> node's receiver (async process - we don't wait for it to hit the disk
> there), and then written onto the disk of self().
>
> You may see HDFS classes BlockReceiver and PacketReceiver in trunk for
> more information.
>
> On Wed, Dec 19, 2012 at 2:26 AM, Jay Vyas <[EMAIL PROTECTED]> wrote:
> > Hi :
> >
> > Do data nodes, when they receive write requests, forward the request to
> the
> > next datanode in the chain after appending to a file? Or rather, do the
> > write and forward asynchronously and at the same time?
> >
> > --
> > Jay Vyas
> > http://jayunit100.blogspot.com
>
>
>
> --
> Harsh J
>

Can't speak for their explanation (not sure what doc you seem to
be referring to?) but the behavior of mirror-first-disk-next has been
there since 0.20 at least - so yes, 1.0 too has the same behavior.

If you speak of the doc
http://hadoop.apache.org/docs/stable/hdfs_design.html#Replication+Pipelining,
that seems to be a bit outdated overall. We could change the text -
mind filing a JIRA and doing so?

On Wed, Dec 19, 2012 at 3:49 AM, Chris Embree <[EMAIL PROTECTED]> wrote:
> Harsh,
>
> Is that a change from 1.0 code?  Hortonworks explains it a little
> differently.
>
> Thanks for the details and pointer to the code.
>
> Chris (another one)
>
> On Dec 18, 2012 5:14 PM, "Harsh J" <[EMAIL PROTECTED]> wrote:
>>
>> Hi,
>>
>> The received write packet is directly socket-written to the next
>> node's receiver (async process - we don't wait for it to hit the disk
>> there), and then written onto the disk of self().
>>
>> You may see HDFS classes BlockReceiver and PacketReceiver in trunk for
>> more information.
>>
>> On Wed, Dec 19, 2012 at 2:26 AM, Jay Vyas <[EMAIL PROTECTED]> wrote:
>> > Hi :
>> >
>> > Do data nodes, when they receive write requests, forward the request to
>> > the
>> > next datanode in the chain after appending to a file? Or rather, do the
>> > write and forward asynchronously and at the same time?
>> >
>> > --
>> > Jay Vyas
>> > http://jayunit100.blogspot.com
>>
>>
>>
>> --
>> Harsh J

--
Harsh J

Hi Jay,

We'll need "real developer" expertise on this, but my understanding of the
documentation is:

Client talks to Name node to get Node/Block assignments,  client then talks
to node 1: write, fwd, node 2: write, fwd, node 3: write, ack node 2, node
2: ack node1, node 1: ack Client and Name Node.

write includes data write and checksum, ack = report success.  The fwd
requests happen AFTER successful write and checksum.

Hope this helps.
Chris

On Tue, Dec 18, 2012 at 3:56 PM, Jay Vyas <[EMAIL PROTECTED]> wrote:

> Hi :
>
> Do data nodes, when they receive write requests, forward the request to
> the next datanode in the chain after appending to a file? Or rather, do the
> write and forward asynchronously and at the same time?
>
> --
> Jay Vyas
> http://jayunit100.blogspot.com
>