The real kicker is going to be memory consumption of one or both of these
services. The NN in particular uses a large amount of RAM to store the
filesystem image. I think that those who are suggesting a breakeven point of
<= 10 nodes are lowballing. In practice, unless your machines are really
thin on the RAM (e.g., 2--4 GB), I haven't seen any cases where these
services need to be separated below the 20-node mark; I've also seen several
clusters of 40 nodes running fine with these services colocated. It depends
on how many files are in HDFS and how frequently you're submitting a lot of
concurrent jobs to MapReduce.
If you're setting up a production environment that you plan to expand,
however, as a best practice you should configure the master node to have two
hostnames (e.g., "nn" and "jt") so that you can have separate hostnames in
fs.default.name and mapred.job.tracker; when the day comes that these
services are placed on different nodes, you'll then be able to just move one
of the hostnames over and not need to reconfigure all 20--40 other nodes.
On Thu, Nov 26, 2009 at 8:27 PM, Srigurunath Chakravarthi <
[EMAIL PROTECTED]> wrote:
> Load wise, it should be very safe to run both JT and NN on a single node
> for small clusters (< 40 Task Trackers and/or Data Nodes). They don't use
> much CPU as such.
> This may even work for larger clusters depending on the type of hardware
> you have and the Hadoop job mix. We usually observe < 5% CPU load with ~80
> DNs/TTs on an 8-code Intel processor based box with 16GB RAM.
> It is best that you observe CPU & mem load on the JT+NN node to take a
> call on whether to separate them. iostat, top or sar should tell you.
> >-----Original Message-----
> >From: John Martyniak [mailto:[EMAIL PROTECTED]]
> >Sent: Friday, November 27, 2009 3:06 AM
> >To: [EMAIL PROTECTED]
> >Cc: <[EMAIL PROTECTED]>
> >Subject: Re: Good idea to run NameNode and JobTracker on same machine?
> >I have a cluster of 4 machines plus one machine to run nn & jt. I
> >have heard that 5 or 6 is the magic #. I will see when I add the next
> >batch of machines.
> >And it seems to running fine.
> >On Nov 26, 2009, at 11:38 AM, Yongqiang He <[EMAIL PROTECTED]>
> >> I think it is definitely not a good idea to combine these two in
> >> production
> >> environment.
> >> Thanks
> >> Yongqiang
> >> On 11/26/09 6:26 AM, "Raymond Jennings III" <[EMAIL PROTECTED]>
> >> wrote:
> >>> Do people normally combine these two processes onto one machine?
> >>> Currently I
> >>> have them on separate machines but I am wondering they use that
> >>> much CPU
> >>> processing time and maybe I should combine them and create another
> >>> DataNode.