Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
MapReduce >> mail # user >> Performance issue in CDH4.0.1 with 32 core machine.


Copy link to this message
-
Performance issue in CDH4.0.1 with 32 core machine.
Hi Guys,

We are running hadoop cluster with Intel xeon 32 core machine, 64 GB Ram,
8x3TB HDD, we are using CDH4.0.1, we regularly monitoring machine, we have
configured 30 mappers per node, When a run a job initialized the mapper
will start assigned 30 mappers, But 10 mapper only utilizing actual cpu,
other are there idle. waiting for something. But successfully completed
taking time to complete the job.

In my understating: hadoop mapper assign posibilites.

1. The machine low become low
2. Data avilable in perticular machine in HDFS
3. Mapper can easily get [Particular job related data shortest way get data
 ]

why some of the core idle?
why 32 core start working 5 to 10 mapper?

we verified with top command, After verified with same job with profiling
machine, profiling for particular mapper process we got ticket spinlock.
will happen 50 to 75%

Any tool available for monitoring hadoop, we already having ganglia,
opereating system= ubuntu 12.04.2 LTS
kernel=3.2.0-37-generic

Note:

My point Machines core not fully utilization happen with hadoop. Some what
related with hardware or Hadoop configuration running parameters

Please guide me.

-Dhanasekaran.
Did I learn something today? If not, I wasted it.
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB