Hi ,

I have 10 machines to build a small cluster, how to build the topological graph between these machines?

I mean how to build the rack use 10 machines.

Thanks !

-Rock

________________________________
The information and any attached documents contained in this message
may be confidential and/or legally privileged. The message is
intended solely for the addressee(s). If you are not the intended
recipient, you are hereby notified that any use, dissemination, or
reproduction is strictly prohibited and may be unlawful. If you are
not the intended recipient, please contact the sender immediately by
return e-mail and destroy all copies of the original message.

Hi Rock,

you mean rack awareness.
http://hadoop.apache.org/common/docs/r0.17.2/hdfs_user_guide.html#Rack+Awareness

Here you find examples:
http://wiki.apache.org/hadoop/topology_rack_awareness_scripts

best,
 Alex
--
Alexander Lorenz
http://mapredit.blogspot.com

On Feb 3, 2012, at 8:03 AM, Jinyan Xu wrote:

> Hi ,
>
> I have 10 machines to build a small cluster, how to build the topological graph between these machines?
>
> I mean how to build the rack use 10 machines.
>
> Thanks !
>
> -Rock
>
> ________________________________
> The information and any attached documents contained in this message
> may be confidential and/or legally privileged. The message is
> intended solely for the addressee(s). If you are not the intended
> recipient, you are hereby notified that any use, dissemination, or
> reproduction is strictly prohibited and may be unlawful. If you are
> not the intended recipient, please contact the sender immediately by
> return e-mail and destroy all copies of the original message.

Hi Alex, sorry for my fuzzy description.

I mean the physical link(network topological structure) for 10 machines.

Thank you!

-----Original Message-----
From: alo alt [mailto:[EMAIL PROTECTED]]
Sent: 2012年2月3日 15:56
To: [EMAIL PROTECTED]
Subject: Re: 10 nodes how to build the topological graph

Hi Rock,

you mean rack awareness.
http://hadoop.apache.org/common/docs/r0.17.2/hdfs_user_guide.html#Rack+Awareness

Here you find examples:
http://wiki.apache.org/hadoop/topology_rack_awareness_scripts

best,
 Alex
--
Alexander Lorenz
http://mapredit.blogspot.com

On Feb 3, 2012, at 8:03 AM, Jinyan Xu wrote:

> Hi ,
>
> I have 10 machines to build a small cluster, how to build the topological graph between these machines?
>
> I mean how to build the rack use 10 machines.
>
> Thanks !
>
> -Rock
>
> ________________________________
> The information and any attached documents contained in this message
> may be confidential and/or legally privileged. The message is
> intended solely for the addressee(s). If you are not the intended
> recipient, you are hereby notified that any use, dissemination, or
> reproduction is strictly prohibited and may be unlawful. If you are
> not the intended recipient, please contact the sender immediately by
> return e-mail and destroy all copies of the original message.
The information and any attached documents contained in this message
may be confidential and/or legally privileged.  The message is
intended solely for the addressee(s).  If you are not the intended
recipient, you are hereby notified that any use, dissemination, or
reproduction is strictly prohibited and may be unlawful.  If you are
not the intended recipient, please contact the sender immediately by
return e-mail and destroy all copies of the original message.

With 10 machines, if you want, you can just build 1 rack. What you might
loose is rack awareness of Hadoop. Meaning all your data will be in one
rack, so a rack lose is cluster loss.

Network topology depends on the following factors:
1. Your replication factor requirement for HDFS blocks
2. How much concurrency you want for Map task w.r.t failures, power and
disk lose.
3. Also, your workloads plays a critical component - how much data is
shuffled.

Building a rack costs you money. Since you have only 10 machines, you can
have one rack with a rack switch connected to all the nodes. My suggestion
would be to take a phased approach: build - observe and iterate for your
workload.

-Bharath
2012/2/3 Jinyan Xu <[EMAIL PROTECTED]>

> Hi Alex, sorry for my fuzzy description.
>
> I mean the physical link(network topological structure) for 10 machines.
>
> Thank you!
>
> -----Original Message-----
> From: alo alt [mailto:[EMAIL PROTECTED]]
> Sent: 2012年2月3日 15:56
> To: [EMAIL PROTECTED]
> Subject: Re: 10 nodes how to build the topological graph
>
> Hi Rock,
>
> you mean rack awareness.
>
> http://hadoop.apache.org/common/docs/r0.17.2/hdfs_user_guide.html#Rack+Awareness
>
> Here you find examples:
> http://wiki.apache.org/hadoop/topology_rack_awareness_scripts
>
> best,
>  Alex
>
>
> --
> Alexander Lorenz
> http://mapredit.blogspot.com
>
> On Feb 3, 2012, at 8:03 AM, Jinyan Xu wrote:
>
> > Hi ,
> >
> > I have 10 machines to build a small cluster, how to build the
> topological graph between these machines?
> >
> > I mean how to build the rack use 10 machines.
> >
> > Thanks !
> >
> > -Rock
> >
> > ________________________________
> > The information and any attached documents contained in this message
> > may be confidential and/or legally privileged. The message is
> > intended solely for the addressee(s). If you are not the intended
> > recipient, you are hereby notified that any use, dissemination, or
> > reproduction is strictly prohibited and may be unlawful. If you are
> > not the intended recipient, please contact the sender immediately by
> > return e-mail and destroy all copies of the original message.
>
>
> The information and any attached documents contained in this message
> may be confidential and/or legally privileged.  The message is
> intended solely for the addressee(s).  If you are not the intended
> recipient, you are hereby notified that any use, dissemination, or
> reproduction is strictly prohibited and may be unlawful.  If you are
> not the intended recipient, please contact the sender immediately by
> return e-mail and destroy all copies of the original message.
>