Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
MapReduce, mail # user - Re: TaskTracker slow to start/join


Copy link to this message
-
Re: TaskTracker slow to start/join
Harsh J 2012-12-03, 04:25
Hi,

This is cause of the TT's behavior of deleting the mapred.local.dir
contents every time you restart it. In your version, 1.0.3, that
process is synchronous and hence it appears like the TT hangs when
there's a lot of data to purge out from those dirs.

On Mon, Dec 3, 2012 at 7:11 AM, Sean McNamara
<[EMAIL PROTECTED]> wrote:
> I have a TaskTracker on a particular node that is very slow to join the
> jobtracker.  When I start it up with ./hadoop-daemon.sh start tasktracker I
> see the daemon fire up and running in top.  The TaskTracker daemon will sit
> there using 50% cpu according to top.   If it helps any this cluster is on
> hadoop 1.0.3.  Does anyone know what the TT could be up to?
>
> Here is the log output:
>
> 2012-12-03 01:28:17,310 INFO org.apache.hadoop.metrics2.impl.MetricsConfig:
> loaded properties from hadoop-metrics2.properties
> 2012-12-03 01:28:17,320 INFO
> org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source
> MetricsSystem,sub=Stats registered.
> 2012-12-03 01:28:17,321 INFO
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot period
> at 10 second(s).
> 2012-12-03 01:28:17,321 INFO
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: TaskTracker metrics
> system started
> 2012-12-03 01:28:17,515 INFO
> org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source ugi
> registered.
> 2012-12-03 01:28:17,679 INFO org.mortbay.log: Logging to
> org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via
> org.mortbay.log.Slf4jLog
> 2012-12-03 01:28:17,726 INFO org.apache.hadoop.http.HttpServer: Added global
> filtersafety (class=org.apache.hadoop.http.HttpServer$QuotingInputFilter)
> 2012-12-03 01:28:17,728 INFO org.apache.hadoop.http.HttpServer: Added filter
> static_user_filter
> (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to
> context WepAppsContext
> 2012-12-03 01:28:17,728 INFO org.apache.hadoop.http.HttpServer: Added filter
> static_user_filter
> (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to
> context static
> 2012-12-03 01:28:17,728 INFO org.apache.hadoop.http.HttpServer: Added filter
> static_user_filter
> (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to
> context logs
> 2012-12-03 01:28:17,747 INFO org.apache.hadoop.mapred.TaskLogsTruncater:
> Initializing logs' truncater with mapRetainSize=-1 and reduceRetainSize=-1
> 2012-12-03 01:28:17,751 INFO org.apache.hadoop.mapred.TaskTracker: Starting
> tasktracker with owner as hadoop
> 2012-12-03 01:28:17,752 INFO org.apache.hadoop.mapred.TaskTracker: Good
> mapred local directories are:
> /mnt/hdfs/data1/mapred,/mnt/hdfs/data2/mapred,/mnt/hdfs/data3/mapred,/mnt/hdfs/data4/mapred,…
>
> It takes 10+ minutes to get past this last line, and then it finally
> continues on and registers in ok with the JT.
>
>
> Thanks

--
Harsh J