Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Hadoop >> mail # general >> rack awareness help


+
Mag Gam 2010-03-03, 03:43
+
Jeff Hammerbacher 2010-03-03, 05:37
+
Mag Gam 2010-03-03, 12:11
+
Allen Wittenauer 2010-03-03, 16:57
+
Neil Bliss 2010-03-03, 19:09
+
Mag Gam 2010-03-04, 01:01
+
Michael Thomas 2010-03-04, 01:09
+
Allen Wittenauer 2010-03-04, 05:21
+
Steve Loughran 2010-03-04, 14:21
+
Mag Gam 2010-03-04, 23:39
Copy link to this message
-
Re: rack awareness help
Hey Mag,

Glad you have solved the problem. I've created a JIRA ticket to improve the
existing documentation: https://issues.apache.org/jira/browse/HADOOP-6616.
If you have some time, it would be useful to hear what could be added to the
existing documentation that would have helped you figure this out sooner.

Thanks,
Jeff

On Thu, Mar 4, 2010 at 3:39 PM, Mag Gam <[EMAIL PROTECTED]> wrote:

> Thanks everyone for explaining this to me instead of giving me RTFM!
>
> I will play around with it and see how far I get.
>
>
>
> On Thu, Mar 4, 2010 at 9:21 AM, Steve Loughran <[EMAIL PROTECTED]> wrote:
> > Allen Wittenauer wrote:
> >>
> >> On 3/3/10 5:01 PM, "Mag Gam" <[EMAIL PROTECTED]> wrote:
> >>
> >>> Thanks Alan! Your presentation is very nice!
> >>
> >> Thanks. :)
> >>
> >>> "If you don't provide a script for rack awareness, it treats every
> >>> node as if it was its own rack". I am using the default settings and
> >>> the report still says only 1 rack.
> >>
> >> Let's take a different approach to convince you. :)
> >>
> >> Think about the question:  Is there a difference between all nodes in
> one
> >> rack vs. every node acting as a lone rack?
> >>
> >> The answer is no, there isn't any difference.  In both cases, all copies
> >> of
> >> the blocks can go to pretty much any node. When a MR job runs, every
> node
> >> would either be considered 'off rack' or 'rack-local'.
> >>
> >> So there is no difference.
> >>
> >>
> >>> Do you mind sharing a script with us on how you determine a rack? and
> >>> a sample <configuration> </configuration> syntax?
> >>
> >> Michael has already posted his, so I'll skip this one. :)
> >>
> >
> > Think Mag probably wanted a shell script.
> >
> > Mag, give your machines IPv4 addresses that map to rack number. 10.1.1.*
> for
> > rack one, 10.1.2.* for rack 2, etc. Then just filter out the IP address
> by
> > the top bytes, returning "10.1.1" for everything in rack one, "10.1.2"
> for
> > rack 2; Hadoop will be happy
> >
>
+
Mag Gam 2010-03-18, 03:34
+
Christopher Tubbs 2010-03-18, 07:15
+
Mag Gam 2010-03-19, 00:41
+
Michael Thomas 2010-03-19, 01:11
+
Christopher Tubbs 2010-03-19, 05:00
+
Mag Gam 2010-03-19, 11:32
+
Christopher Tubbs 2010-03-19, 13:36
+
Allen Wittenauer 2010-03-19, 15:13
+
Mag Gam 2010-03-20, 12:55
+
Christopher Tubbs 2010-03-20, 13:46
+
Michael Thomas 2010-03-20, 16:15
+
Michael Thomas 2010-03-19, 01:21
+
Michael Thomas 2010-03-19, 01:26
+
Allen Wittenauer 2010-03-03, 16:55
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB