Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
HDFS, mail # dev - [Discuss] Merge federation branch HDFS-1052 into trunk


+
Suresh Srinivas 2011-04-22, 16:48
+
Dhruba Borthakur 2011-04-23, 08:08
+
Doug Cutting 2011-04-25, 21:36
+
suresh srinivas 2011-04-26, 17:29
+
suresh srinivas 2011-04-26, 23:06
+
Doug Cutting 2011-04-27, 04:43
+
Konstantin Shvachko 2011-04-27, 05:26
+
suresh srinivas 2011-04-27, 06:34
+
suresh srinivas 2011-04-27, 06:55
+
suresh srinivas 2011-04-27, 17:02
+
Tsz Wo \ 2011-04-27, 17:08
+
Devaraj Das 2011-04-27, 17:08
+
Konstantin Boudnik 2011-04-27, 17:41
+
suresh srinivas 2011-04-27, 21:36
+
Konstantin Shvachko 2011-04-28, 05:18
+
Hairong 2011-04-27, 17:46
+
Konstantin Shvachko 2011-04-28, 04:56
+
Konstantin Boudnik 2011-04-28, 13:36
+
suresh srinivas 2011-04-28, 18:02
+
Owen OMalley 2011-04-27, 20:53
+
suresh srinivas 2011-04-27, 21:44
+
Konstantin Shvachko 2011-04-28, 05:12
+
Owen OMalley 2011-04-28, 20:33
+
suresh srinivas 2011-04-28, 22:12
+
Konstantin Shvachko 2011-04-29, 06:30
+
Todd Lipcon 2011-05-02, 21:44
+
suresh srinivas 2011-05-03, 02:17
+
Sanjay Radia 2011-04-27, 00:26
+
Konstantin Boudnik 2011-04-27, 00:59
+
Dhruba Borthakur 2011-04-27, 04:27
+
Tsz Wo \ 2011-04-27, 05:16
Copy link to this message
-
Re: [Discuss] Merge federation branch HDFS-1052 into trunk
Konstantin Shvachko 2011-04-27, 05:36
Dhruba,

It would be very valuable for the community to share your experience
if you performed any independent testing of the federation branch.

Thanks,
--Konstantin

On Tue, Apr 26, 2011 at 9:27 PM, Dhruba Borthakur <[EMAIL PROTECTED]> wrote:

> I feel that making the datanode talk to multiple namenodes is very
> valuable,
> especially when there is plenty of storage available on a single datanode
> machine (think 24 TB to 36 TB) and a single namenode does not have enough
> memory to hold all file metadata for such a large cluster in memory.
>
> This is a feature that we are in dire need of, and could put it to good use
> starting "yesterday"!
>
> thanks,
> dhruba
>
> On Tue, Apr 26, 2011 at 5:59 PM, Konstantin Boudnik <[EMAIL PROTECTED]>
> wrote:
>
> > Sanjay,
> >
> > I assume the outlined changes won't an earlier version of HDFS from
> > upgrads to the federation version, right?
> >
> > Cos
> >
> > On Tue, Apr 26, 2011 at 17:26, Sanjay Radia <[EMAIL PROTECTED]>
> wrote:
> > >
> > > Changes to the code base
> > >  - The fundamental code change is to extend the notion of block id to
> now
> > > include a block pool id.
> > > - The  NN had little change, the protocols did change to include the
> > block
> > > pool id.
> > > - The DN code did change. Each data structure is now indexed by the
> block
> > > pool id -- while this is a code change, it is architecturally very
> simple
> > > and low risk.
> > > - We also did a fair amount of cleanup of threads used to send block
> > reports
> > > - while it was not strictly necessary to do the cleanup we took the
> extra
> > > effort to pay the technical debt. As Dhruba recently noted, adding
> > support
> > > to send block reports to primary and secondary NN for HA will be now
> much
> > > easier to do.
> >
>
>
>
> --
> Connect to me at http://www.facebook.com/dhruba
>
+
Konstantin Boudnik 2011-04-27, 05:40
+
suresh srinivas 2011-04-27, 06:28
+
Sanjay Radia 2011-04-27, 14:03
+
Eli Collins 2011-04-27, 21:36
+
suresh srinivas 2011-04-28, 00:22