Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Hadoop >> mail # general >> Hadoop Java Versions


+
Allen Wittenauer 2011-06-15, 00:16
+
Scott Carey 2011-06-22, 20:27
+
Allen Wittenauer 2011-06-22, 20:49
+
Scott Carey 2011-06-23, 02:42
+
Steve Loughran 2011-06-23, 12:49
+
Scott Carey 2011-06-26, 19:23
+
Steve Loughran 2011-06-27, 11:38
+
Ryan Rawson 2011-06-28, 00:10
Copy link to this message
-
Re: Hadoop Java Versions
Come to Srivas talk at the Summit.

On Mon, Jun 27, 2011 at 5:10 PM, Ryan Rawson <[EMAIL PROTECTED]> wrote:

> On the subject of gige vs 10-gige, I think that we will very shortly
> be seeing interest in 10gig, since gige is only 120MB/sec - 1 hard
> drive of streaming data.  Nodes with 4+ disks are throttled by the
> network.  On a small cluster (20 nodes), the replication traffic can
> choke a cluster to death.  The only way to fix quickly it is to bring
> that node back up.  Perhaps the HortonWorks guys can work on that.
>
> -ryan
>
> On Mon, Jun 27, 2011 at 4:38 AM, Steve Loughran <[EMAIL PROTECTED]> wrote:
> > On 26/06/11 20:23, Scott Carey wrote:
> >>
> >>
> >> On 6/23/11 5:49 AM, "Steve Loughran"<[EMAIL PROTECTED]>  wrote:
> >>
> >
> >>> what's your HW setup? #cores/server, #servers, underlying OS?
> >>
> >> CentOS 5.6.
> >> 4 cores / 8 threads a server (Nehalem generation Intel processor).
> >
> >
> > that should be enough to find problems. I've just moved up to a 6-core 12
> > thread desktop and that found problems on some non-Hadoop code, which
> shows
> > that the more threads you have, and the faster the machines are, the more
> > your race conditions show up. With Hadoop the fact that you can have
> 10-1000
> > servers means that in a large cluster the probability of that race
> condition
> > showing up scales well.
> >
> >> Also run a smaller cluster with 2x quad core Core 2 generation Xeons.
> >>
> >> Off topic:
> >> The single proc Nehalem is faster than the dual core 2's for most use
> >> cases -- and much lower power.  Looking forward to single proc 4 or 6
> core
> >> Sandy Bridge based systems for the next expansion -- testing 4 core vs 4
> >> core has these 30% faster than the Nehalem generation systems in CPU
> bound
> >> tasks and lower power.  Intel prices single socket Xeons so much lower
> >> than the Dual socket ones that the best value for us is to get more
> single
> >> socket servers rather than fewer dual socket ones (with similar
> processor
> >> to hard drive ratio).
> >
> > Yes, in a large cluster the price of filling the second socket can
> compare
> > to a lot of storage, and TB of storage is more tangible. I guess it
> depends
> > on your application.
> >
> > Regarding Sandy Bridge, I've no experience of those, but I worry that 10
> > Gbps is still bleeding edge, and shouldn't be needed for code with good
> > locality anyway; it is probably more cost effective to stay at
> 1Gbps/server,
> > though the issue there is the #of HDD/s server generates lots of
> replication
> > traffic when a single server fails...
> >
>
+
Segel, Mike 2011-06-28, 01:54
+
Ryan Rawson 2011-06-28, 02:33
+
Segel, Mike 2011-06-28, 03:49
+
Steve Loughran 2011-06-28, 09:59
+
Michel Segel 2011-06-28, 12:27
+
Arun C Murthy 2011-06-28, 17:25
+
Evert Lammerts 2011-06-30, 21:31
+
Evert Lammerts 2011-06-30, 21:37
+
Aaron Eng 2011-06-30, 23:18
+
Ted Dunning 2011-07-01, 00:16
+
Todd Lipcon 2011-07-01, 00:24
+
Ted Dunning 2011-07-01, 01:12
+
M. C. Srivas 2011-07-01, 04:08
+
Ian Holsman 2011-07-01, 04:47
+
M. C. Srivas 2011-07-01, 05:06
+
Ted Dunning 2011-07-01, 05:09
+
Scott Carey 2011-07-01, 17:22
+
Abhishek Mehta 2011-07-01, 18:53
+
Ted Dunning 2011-07-01, 21:38
+
Ian Holsman 2011-07-03, 04:16
+
Eric Baldeschwieler 2011-07-13, 14:59
+
Ted Dunning 2011-07-13, 15:48
+
Steve Loughran 2011-07-01, 16:12
+
Ted Dunning 2011-07-01, 18:09
+
Scott Carey 2011-06-28, 02:58
+
Evert Lammerts 2011-06-30, 21:40
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB