Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Hadoop >> mail # user >> 1gig or 10gig network for cluster?


+
Koert Kuipers 2011-12-23, 17:23
+
alo alt 2011-12-23, 18:44
+
Mads Toftum 2011-12-23, 19:15
Copy link to this message
-
Re: 1gig or 10gig network for cluster?
One or two 1gig nics on a 10g backbone sound reasonable with "only" 4 1T drives. 12*2T disks per node are getting more common and do not all have 10gig network cards, even on 600+ node clusters.

Cheers,

Joep

Sent from my iPhone

On Dec 23, 2011, at 11:15 AM, Mads Toftum <[EMAIL PROTECTED]> wrote:

> On Fri, Dec 23, 2011 at 12:23:59PM -0500, Koert Kuipers wrote:
>> For a hadoop cluster that starts medium size (50 nodes) but could grow to
>> hundred of nodes, what is the recommended network in the rack? 1gig or 10gig
>> We have machines with 8 cores, 4 X 1tb drive (could grow to 8 X 1b drive),
>> 48 Gb ram per node.
>> We expect "balanced" usage of the cluster (both storage and computations).
>
> If it were me, I'd start by looking at the amount of data you need to
> handle. Guestimate on both peak and average size. For simplification, start by
> guessing whether you'll be reading or writing more. If writes will be
> the driver, then you need to adjust for the number of copies. Or if you
> don't know how much data, then you could do a read and write test on one
> of the machines and figure out how much it can produce and go from that.
> I haven't looked at the pricing of 10gig gear lately, but I wouldn't be
> surprised if it made more sense for you to run simple nodes with a
> couple of 1gig cards and your infrastructure / backbone on 10gig.
> There's really too much ymmv based on your specific case, so that you'll
> only really know if you test on a handfull of machines.
>
> vh
>
> Mads Toftum
> --
> http://soulfood.dk
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB