Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Hadoop, mail # general - Hadoop Java Versions


+
Allen Wittenauer 2011-06-15, 00:16
+
Scott Carey 2011-06-22, 20:27
+
Allen Wittenauer 2011-06-22, 20:49
+
Scott Carey 2011-06-23, 02:42
+
Steve Loughran 2011-06-23, 12:49
+
Scott Carey 2011-06-26, 19:23
+
Steve Loughran 2011-06-27, 11:38
+
Ryan Rawson 2011-06-28, 00:10
Copy link to this message
-
Re: Hadoop Java Versions
Ted Dunning 2011-06-28, 00:12
Come to Srivas talk at the Summit.

On Mon, Jun 27, 2011 at 5:10 PM, Ryan Rawson <[EMAIL PROTECTED]> wrote:

> On the subject of gige vs 10-gige, I think that we will very shortly
> be seeing interest in 10gig, since gige is only 120MB/sec - 1 hard
> drive of streaming data.  Nodes with 4+ disks are throttled by the
> network.  On a small cluster (20 nodes), the replication traffic can
> choke a cluster to death.  The only way to fix quickly it is to bring
> that node back up.  Perhaps the HortonWorks guys can work on that.
>
> -ryan
>
> On Mon, Jun 27, 2011 at 4:38 AM, Steve Loughran <[EMAIL PROTECTED]> wrote:
> > On 26/06/11 20:23, Scott Carey wrote:
> >>
> >>
> >> On 6/23/11 5:49 AM, "Steve Loughran"<[EMAIL PROTECTED]>  wrote:
> >>
> >
> >>> what's your HW setup? #cores/server, #servers, underlying OS?
> >>
> >> CentOS 5.6.
> >> 4 cores / 8 threads a server (Nehalem generation Intel processor).
> >
> >
> > that should be enough to find problems. I've just moved up to a 6-core 12
> > thread desktop and that found problems on some non-Hadoop code, which
> shows
> > that the more threads you have, and the faster the machines are, the more
> > your race conditions show up. With Hadoop the fact that you can have
> 10-1000
> > servers means that in a large cluster the probability of that race
> condition
> > showing up scales well.
> >
> >> Also run a smaller cluster with 2x quad core Core 2 generation Xeons.
> >>
> >> Off topic:
> >> The single proc Nehalem is faster than the dual core 2's for most use
> >> cases -- and much lower power.  Looking forward to single proc 4 or 6
> core
> >> Sandy Bridge based systems for the next expansion -- testing 4 core vs 4
> >> core has these 30% faster than the Nehalem generation systems in CPU
> bound
> >> tasks and lower power.  Intel prices single socket Xeons so much lower
> >> than the Dual socket ones that the best value for us is to get more
> single
> >> socket servers rather than fewer dual socket ones (with similar
> processor
> >> to hard drive ratio).
> >
> > Yes, in a large cluster the price of filling the second socket can
> compare
> > to a lot of storage, and TB of storage is more tangible. I guess it
> depends
> > on your application.
> >
> > Regarding Sandy Bridge, I've no experience of those, but I worry that 10
> > Gbps is still bleeding edge, and shouldn't be needed for code with good
> > locality anyway; it is probably more cost effective to stay at
> 1Gbps/server,
> > though the issue there is the #of HDD/s server generates lots of
> replication
> > traffic when a single server fails...
> >
>
+
Segel, Mike 2011-06-28, 01:54
+
Ryan Rawson 2011-06-28, 02:33
+
Segel, Mike 2011-06-28, 03:49
+
Steve Loughran 2011-06-28, 09:59
+
Michel Segel 2011-06-28, 12:27
+
Arun C Murthy 2011-06-28, 17:25
+
Evert Lammerts 2011-06-30, 21:31
+
Evert Lammerts 2011-06-30, 21:37
+
Aaron Eng 2011-06-30, 23:18
+
Ted Dunning 2011-07-01, 00:16
+
Todd Lipcon 2011-07-01, 00:24
+
Ted Dunning 2011-07-01, 01:12
+
M. C. Srivas 2011-07-01, 04:08
+
Ian Holsman 2011-07-01, 04:47
+
M. C. Srivas 2011-07-01, 05:06
+
Ted Dunning 2011-07-01, 05:09
+
Scott Carey 2011-07-01, 17:22
+
Abhishek Mehta 2011-07-01, 18:53
+
Ted Dunning 2011-07-01, 21:38
+
Ian Holsman 2011-07-03, 04:16
+
Eric Baldeschwieler 2011-07-13, 14:59
+
Ted Dunning 2011-07-13, 15:48
+
Steve Loughran 2011-07-01, 16:12
+
Ted Dunning 2011-07-01, 18:09
+
Scott Carey 2011-06-28, 02:58
+
Evert Lammerts 2011-06-30, 21:40