Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hadoop >> mail # dev >> How HDFS decides where to put the block


Copy link to this message
-
Re: How HDFS decides where to put the block
Yes, that's it

Thanks

Nan

On Mon, Apr 18, 2011 at 10:16 PM, Harsh J <[EMAIL PROTECTED]> wrote:

> Hello,
>
> On Mon, Apr 18, 2011 at 7:16 PM, Nan Zhu <[EMAIL PROTECTED]> wrote:
> > Hi, all
> >
> > I'm confused by a question that "how does the HDFS decide where to put
> the
> > data blocks "
> >
> > I mean that the user invokes some commands like "./hadoop put ***", we
> > assume that this  file consistes of 3 blocks, but how HDFS decides where
> > these 3 blocks to be put?
> >
> > Most of the materials don't involve this issue, but just introduce the
> data
> > replica where talking about blocks in HDFS,
> >
>
> I'm guessing you're looking for the BlockPlacementPolicy
> implementations [1] and how it is applied in the HDFS.
>
> Basically, the NameNode chooses the set of DNs for every new-block
> request (from a client) using this policy, and the DFSClient gets a
> list of all the nodes. It goes on to pick the first one among them to
> write the data to. The replication happens async, later.
>
> [1] - BlockPolicyPlacementDefault is the default implementation in
> use. It's source available in the o.a.h.hdfs.server.namenode package.
>
> --
> Harsh J
>

--
Nan Zhu
School of Software,5501
Shanghai Jiao Tong University
800,Dongchuan Road,Shanghai,China
E-Mail: [EMAIL PROTECTED]
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB