Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Hadoop, mail # user - Re: Which hardware to choose


Copy link to this message
-
Re: Which hardware to choose
Oleg Ruchovets 2012-10-02, 14:59
Great ,

Thank you for the such detailed information,

By the way what type of Disk Controller do you use?

Thanks
Oleg.
On Tue, Oct 2, 2012 at 6:34 AM, Alexander Pivovarov <[EMAIL PROTECTED]>wrote:

> Privet Oleg
>
> Cloudera and Dell setup the following cluster for my company
> Company receives 1.5 TB raw data per day
>
> 38 data nodes + 2 Name Nodes
>
> Data Node:
> Dell PowerEdge C2100 series
> 2 x XEON x5670
> 48 GB RAM ECC  (12x4GB 1333MHz)
> 12 x 2 TB  7200 RPM SATA HDD (with hot swap)  JBOD
> Intel Gigabit ET Dual port PCIe x4
> Redundant Power Supply
> Hadoop CDH3
> max map tasks 24
> max reduce tasks 8
>
> Name Node and Secondary Name Node are the similar but
> 96GB RAM  (not sure why)
> 6x600Gb 15 RPM Serial SCSI
> RAID10
>
>
> another config is here
> page 298
>
> http://books.google.com/books?id=Wu_xeGdU4G8C&pg=PA298&lpg=PA298&dq=hadoop+jbod&source=bl&ots=i7xVQBPb_w&sig=8mhq-MtpkRcTiRB1ioKciMxIasg&hl=en&sa=X&ei=AGtqUMK6D8T10gHD4ICQAQ&ved=0CEMQ6AEwAg#v=onepage&q=hadoop%20jbod&f=false
>
>
> you probably need just 1 computer with 10 x 2 TB SATA HDD
>
>
>
> On Mon, Oct 1, 2012 at 6:02 PM, Oleg Ruchovets <[EMAIL PROTECTED]>
> wrote:
>
> > Hi ,
> >   We are on a very early stage of our hadoop project and want to do a
> POC.
> >
> > We have ~ 5-6 terabytes of row data and we are going to execute some
> > aggregations.
> >
> > We plan to use  8 - 10 machines
> >
> > Questions:
> >
> >   1)  Which hardware should we use:
> >     a) How many discs , what discs is better to use?
> >     b) How many RAM?
> >     c) How many CPUs?
> >
> >
> >    2) Please share best practices and tips / tricks related to utilise
> > hardware using for hadoop projects.
> >
> > Thanks in advance
> > Oleg.
> >
>
+
Alexander Pivovarov 2012-10-02, 18:55
+
hadoopman 2012-10-03, 01:14
+
Michael Segel 2012-10-03, 01:40
+
Marcos Ortiz 2012-10-03, 01:57
+
Michael Segel 2012-10-03, 02:35
+
Alexander Pivovarov 2012-10-03, 03:42
+
J. Rottinghuis 2012-10-03, 17:09
+
Michael Segel 2012-10-03, 17:21
+
Russell Jurney 2012-10-03, 01:19
+
hadoopman 2012-10-03, 01:33