Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
HDFS >> mail # dev >> [Discuss] Merge federation branch HDFS-1052 into trunk


+
Suresh Srinivas 2011-04-22, 16:48
+
Dhruba Borthakur 2011-04-23, 08:08
Copy link to this message
-
Re: [Discuss] Merge federation branch HDFS-1052 into trunk
On 04/22/2011 09:48 AM, Suresh Srinivas wrote:
> A few weeks ago, I had sent an email about the progress of HDFS
> federation development in HDFS-1052 branch. I am happy to announce
> that all the tasks related to this feature development is complete
> and it is ready to be integrated into trunk.

A couple of questions:

1. Can you please describe the significant advantages this approach has
over a symlink-based approach?

It seems to me that one could run multiple namenodes on separate boxes
and run multile datanode processes per storage box configured with
something like:

first datanode process configuraton
  fs.default.name = hdfs://nn1/
  dfs.data.dir = /drive1/nn1/,drive2/nn1/...

second datanode process configuraton
  fs.default.name = hdfs://nn2/
  dfs.data.dir = /drive1/nn2/,drive2/nn2/...

...

Then symlinks could be used between nn1, nn2, etc to provide a
reasonably unified namespace.  From the benefits listed in the design
document it is not clear to me what the clear, substantial benefits are
over such a configuration.

2. How much testing has been performed on this?  The patch modifies much
of the logic of Hadoop's central component, upon which the performance
and reliability of most other components of the ecosystem depend.  It
seems to me that such an invasive change should be well tested before it
is merged to trunk.  Can you please tell me how this has been tested
beyond unit tests?

Thanks!

Doug
+
suresh srinivas 2011-04-26, 17:29
+
suresh srinivas 2011-04-26, 23:06
+
Doug Cutting 2011-04-27, 04:43
+
Konstantin Shvachko 2011-04-27, 05:26
+
suresh srinivas 2011-04-27, 06:34
+
suresh srinivas 2011-04-27, 06:55
+
suresh srinivas 2011-04-27, 17:02
+
Tsz Wo \ 2011-04-27, 17:08
+
Devaraj Das 2011-04-27, 17:08
+
Konstantin Boudnik 2011-04-27, 17:41
+
suresh srinivas 2011-04-27, 21:36
+
Konstantin Shvachko 2011-04-28, 05:18
+
Hairong 2011-04-27, 17:46
+
Konstantin Shvachko 2011-04-28, 04:56
+
Konstantin Boudnik 2011-04-28, 13:36
+
suresh srinivas 2011-04-28, 18:02
+
Owen OMalley 2011-04-27, 20:53
+
suresh srinivas 2011-04-27, 21:44
+
Konstantin Shvachko 2011-04-28, 05:12
+
Owen OMalley 2011-04-28, 20:33
+
suresh srinivas 2011-04-28, 22:12
+
Konstantin Shvachko 2011-04-29, 06:30
+
Todd Lipcon 2011-05-02, 21:44
+
suresh srinivas 2011-05-03, 02:17
+
Sanjay Radia 2011-04-27, 00:26
+
Konstantin Boudnik 2011-04-27, 00:59
+
Dhruba Borthakur 2011-04-27, 04:27
+
Tsz Wo \ 2011-04-27, 05:16
+
Konstantin Shvachko 2011-04-27, 05:36
+
Konstantin Boudnik 2011-04-27, 05:40
+
suresh srinivas 2011-04-27, 06:28
+
Sanjay Radia 2011-04-27, 14:03
+
Eli Collins 2011-04-27, 21:36
+
suresh srinivas 2011-04-28, 00:22
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB