Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
HBase, mail # user - High OS Load Numbers when idle


Copy link to this message
-
Re: High OS Load Numbers when idle
Cosmin Lehene 2010-08-18, 07:26
George,

Did you try to monitor the CPU in real-time to see any patterns? Does it do any CPU spikes? Even so, the load averages should be matched against the number of cores/hyper threads. How many cores do you see in top? (I personally prefer htop to get a cluster overview in realtime). If you have 2 CPUs with 2 cores each the maximum load you should have would be 4. However if the cluster is idle it should go as low as 0.2 - 0.3.  If you don't see the real-time utilization I'd check if Ganglia is set up correctly.

Cosmin
On Aug 18, 2010, at 1:11 AM, George Stathis wrote:

> No takers? :-) Am I missing something too obvious?
>
> On Tue, Aug 17, 2010 at 2:03 PM, George Stathis <[EMAIL PROTECTED]> wrote:
>
>> Hello,
>>
>> We have just setup a new cluster on EC2 using Hadoop 0.20.2 and HBase
>> 0.20.3. Our small setup as of right now consists of one master and four
>> slaves with a replication factor of 2:
>>
>> Master: xLarge instance with 2 CPUs and 17.5 GB RAM - runs 1 namenode, 1
>> secondarynamenode, 1 jobtracker, 1 hbasemaster, 1 zookeeper (uses its' own
>> dedicated EMS drive)
>> Slaves: xLarge instance with 2 CPUs and 17.5 GB RAM each - run 1 datanode,
>> 1 tasktracker, 1 regionserver
>>
>> We have also installed Ganglia to monitor the cluster stats as we are about
>> to run some performance tests but, right out of the box, we are noticing
>> high system loads (especially on the master node) without any activity
>> happening on the clister. Of course, the CPUs are not being utilized at all,
>> but Ganglia is reporting almost all nodes in the red as the 1, 5 an 15
>> minute load times are all above 100% most of the time (i.e. there are more
>> than two processes at a time competing for the 2 CPUs time).
>>
>> Question1: is this normal?
>> Question2: does it matter since each process barely uses any of the CPU
>> time?
>>
>> Thank you in advance and pardon the noob questions.
>>
>> -GS
>>