Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
HDFS >> mail # user >> HDFS: buffer before contacts Namenode?


Copy link to this message
-
Re: HDFS: buffer before contacts Namenode?
DataNode only buffers a packet before it contacts NameNode for allocating
DataNodes to place the block. The doc you read might be too old.

Hairong

On 8/9/10 7:14 PM, "elton sky" <[EMAIL PROTECTED]> wrote:

> hello folks,
>
> I can see from the design doc of HDFS, says: client will buffer a block size
> worth of data before contacting namenode for data node info. This is a network
> throughput optimal way.
> However, I could not find this buffer processing procedure in source code.
>
> In DFSClient.DataStreamer, it waits for dataqueue to be not empty and starts
> to request namenode and build a pipeline. The number of packets in the
> dataqueue is always 1 when this happens! 
> I am confused here. Can anyone address this if I am wrong?
>

NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB