Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
HDFS >> mail # dev >> [Discuss] Merge federation branch HDFS-1052 into trunk


+
Suresh Srinivas 2011-04-22, 16:48
+
Dhruba Borthakur 2011-04-23, 08:08
+
Doug Cutting 2011-04-25, 21:36
+
suresh srinivas 2011-04-26, 17:29
+
suresh srinivas 2011-04-26, 23:06
+
Doug Cutting 2011-04-27, 04:43
+
Konstantin Shvachko 2011-04-27, 05:26
+
suresh srinivas 2011-04-27, 06:34
+
suresh srinivas 2011-04-27, 06:55
+
suresh srinivas 2011-04-27, 17:02
+
Tsz Wo \ 2011-04-27, 17:08
+
Devaraj Das 2011-04-27, 17:08
+
Konstantin Boudnik 2011-04-27, 17:41
+
suresh srinivas 2011-04-27, 21:36
+
Konstantin Shvachko 2011-04-28, 05:18
+
Hairong 2011-04-27, 17:46
+
Konstantin Shvachko 2011-04-28, 04:56
+
Konstantin Boudnik 2011-04-28, 13:36
+
suresh srinivas 2011-04-28, 18:02
+
Owen OMalley 2011-04-27, 20:53
+
suresh srinivas 2011-04-27, 21:44
+
Konstantin Shvachko 2011-04-28, 05:12
+
Owen OMalley 2011-04-28, 20:33
+
suresh srinivas 2011-04-28, 22:12
+
Konstantin Shvachko 2011-04-29, 06:30
+
Todd Lipcon 2011-05-02, 21:44
+
suresh srinivas 2011-05-03, 02:17
+
Sanjay Radia 2011-04-27, 00:26
+
Konstantin Boudnik 2011-04-27, 00:59
+
Dhruba Borthakur 2011-04-27, 04:27
+
Tsz Wo \ 2011-04-27, 05:16
Copy link to this message
-
Re: [Discuss] Merge federation branch HDFS-1052 into trunk
Dhruba,

It would be very valuable for the community to share your experience
if you performed any independent testing of the federation branch.

Thanks,
--Konstantin

On Tue, Apr 26, 2011 at 9:27 PM, Dhruba Borthakur <[EMAIL PROTECTED]> wrote:

> I feel that making the datanode talk to multiple namenodes is very
> valuable,
> especially when there is plenty of storage available on a single datanode
> machine (think 24 TB to 36 TB) and a single namenode does not have enough
> memory to hold all file metadata for such a large cluster in memory.
>
> This is a feature that we are in dire need of, and could put it to good use
> starting "yesterday"!
>
> thanks,
> dhruba
>
> On Tue, Apr 26, 2011 at 5:59 PM, Konstantin Boudnik <[EMAIL PROTECTED]>
> wrote:
>
> > Sanjay,
> >
> > I assume the outlined changes won't an earlier version of HDFS from
> > upgrads to the federation version, right?
> >
> > Cos
> >
> > On Tue, Apr 26, 2011 at 17:26, Sanjay Radia <[EMAIL PROTECTED]>
> wrote:
> > >
> > > Changes to the code base
> > >  - The fundamental code change is to extend the notion of block id to
> now
> > > include a block pool id.
> > > - The  NN had little change, the protocols did change to include the
> > block
> > > pool id.
> > > - The DN code did change. Each data structure is now indexed by the
> block
> > > pool id -- while this is a code change, it is architecturally very
> simple
> > > and low risk.
> > > - We also did a fair amount of cleanup of threads used to send block
> > reports
> > > - while it was not strictly necessary to do the cleanup we took the
> extra
> > > effort to pay the technical debt. As Dhruba recently noted, adding
> > support
> > > to send block reports to primary and secondary NN for HA will be now
> much
> > > easier to do.
> >
>
>
>
> --
> Connect to me at http://www.facebook.com/dhruba
>
+
Konstantin Boudnik 2011-04-27, 05:40
+
suresh srinivas 2011-04-27, 06:28
+
Sanjay Radia 2011-04-27, 14:03
+
Eli Collins 2011-04-27, 21:36
+
suresh srinivas 2011-04-28, 00:22
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB