Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
HDFS, mail # dev - [Discuss] Merge federation branch HDFS-1052 into trunk


Copy link to this message
-
Re: [Discuss] Merge federation branch HDFS-1052 into trunk
Doug Cutting 2011-04-25, 21:36
On 04/22/2011 09:48 AM, Suresh Srinivas wrote:
> A few weeks ago, I had sent an email about the progress of HDFS
> federation development in HDFS-1052 branch. I am happy to announce
> that all the tasks related to this feature development is complete
> and it is ready to be integrated into trunk.

A couple of questions:

1. Can you please describe the significant advantages this approach has
over a symlink-based approach?

It seems to me that one could run multiple namenodes on separate boxes
and run multile datanode processes per storage box configured with
something like:

first datanode process configuraton
  fs.default.name = hdfs://nn1/
  dfs.data.dir = /drive1/nn1/,drive2/nn1/...

second datanode process configuraton
  fs.default.name = hdfs://nn2/
  dfs.data.dir = /drive1/nn2/,drive2/nn2/...

...

Then symlinks could be used between nn1, nn2, etc to provide a
reasonably unified namespace.  From the benefits listed in the design
document it is not clear to me what the clear, substantial benefits are
over such a configuration.

2. How much testing has been performed on this?  The patch modifies much
of the logic of Hadoop's central component, upon which the performance
and reliability of most other components of the ecosystem depend.  It
seems to me that such an invasive change should be well tested before it
is merged to trunk.  Can you please tell me how this has been tested
beyond unit tests?

Thanks!

Doug