Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
HDFS >> mail # user >> TaskTracker slow to start/join


Copy link to this message
-
TaskTracker slow to start/join
I have a TaskTracker on a particular node that is very slow to join the jobtracker.  When I start it up with ./hadoop-daemon.sh start tasktracker I see the daemon fire up and running in top.  The TaskTracker daemon will sit there using 50% cpu according to top.   If it helps any this cluster is on hadoop 1.0.3.  Does anyone know what the TT could be up to?

Here is the log output:

2012-12-03 01:28:17,310 INFO org.apache.hadoop.metrics2.impl.MetricsConfig: loaded properties from hadoop-metrics2.properties
2012-12-03 01:28:17,320 INFO org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source MetricsSystem,sub=Stats registered.
2012-12-03 01:28:17,321 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot period at 10 second(s).
2012-12-03 01:28:17,321 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: TaskTracker metrics system started
2012-12-03 01:28:17,515 INFO org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source ugi registered.
2012-12-03 01:28:17,679 INFO org.mortbay.log: Logging to org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog
2012-12-03 01:28:17,726 INFO org.apache.hadoop.http.HttpServer: Added global filtersafety (class=org.apache.hadoop.http.HttpServer$QuotingInputFilter)
2012-12-03 01:28:17,728 INFO org.apache.hadoop.http.HttpServer: Added filter static_user_filter (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to context WepAppsContext
2012-12-03 01:28:17,728 INFO org.apache.hadoop.http.HttpServer: Added filter static_user_filter (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to context static
2012-12-03 01:28:17,728 INFO org.apache.hadoop.http.HttpServer: Added filter static_user_filter (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to context logs
2012-12-03 01:28:17,747 INFO org.apache.hadoop.mapred.TaskLogsTruncater: Initializing logs' truncater with mapRetainSize=-1 and reduceRetainSize=-1
2012-12-03 01:28:17,751 INFO org.apache.hadoop.mapred.TaskTracker: Starting tasktracker with owner as hadoop
2012-12-03 01:28:17,752 INFO org.apache.hadoop.mapred.TaskTracker: Good mapred local directories are: /mnt/hdfs/data1/mapred,/mnt/hdfs/data2/mapred,/mnt/hdfs/data3/mapred,/mnt/hdfs/data4/mapred,…

It takes 10+ minutes to get past this last line, and then it finally continues on and registers in ok with the JT.
Thanks