Hi,

This is cause of the TT's behavior of deleting the mapred.local.dir
contents every time you restart it. In your version, 1.0.3, that
process is synchronous and hence it appears like the TT hangs when
there's a lot of data to purge out from those dirs.

On Mon, Dec 3, 2012 at 7:11 AM, Sean McNamara
<[EMAIL PROTECTED]> wrote:
> I have a TaskTracker on a particular node that is very slow to join the
> jobtracker.  When I start it up with ./hadoop-daemon.sh start tasktracker I
> see the daemon fire up and running in top.  The TaskTracker daemon will sit
> there using 50% cpu according to top.   If it helps any this cluster is on
> hadoop 1.0.3.  Does anyone know what the TT could be up to?
>
> Here is the log output:
>
> 2012-12-03 01:28:17,310 INFO org.apache.hadoop.metrics2.impl.MetricsConfig:
> loaded properties from hadoop-metrics2.properties
> 2012-12-03 01:28:17,320 INFO
> org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source
> MetricsSystem,sub=Stats registered.
> 2012-12-03 01:28:17,321 INFO
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot period
> at 10 second(s).
> 2012-12-03 01:28:17,321 INFO
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: TaskTracker metrics
> system started
> 2012-12-03 01:28:17,515 INFO
> org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source ugi
> registered.
> 2012-12-03 01:28:17,679 INFO org.mortbay.log: Logging to
> org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via
> org.mortbay.log.Slf4jLog
> 2012-12-03 01:28:17,726 INFO org.apache.hadoop.http.HttpServer: Added global
> filtersafety (class=org.apache.hadoop.http.HttpServer$QuotingInputFilter)
> 2012-12-03 01:28:17,728 INFO org.apache.hadoop.http.HttpServer: Added filter
> static_user_filter
> (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to
> context WepAppsContext
> 2012-12-03 01:28:17,728 INFO org.apache.hadoop.http.HttpServer: Added filter
> static_user_filter
> (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to
> context static
> 2012-12-03 01:28:17,728 INFO org.apache.hadoop.http.HttpServer: Added filter
> static_user_filter
> (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to
> context logs
> 2012-12-03 01:28:17,747 INFO org.apache.hadoop.mapred.TaskLogsTruncater:
> Initializing logs' truncater with mapRetainSize=-1 and reduceRetainSize=-1
> 2012-12-03 01:28:17,751 INFO org.apache.hadoop.mapred.TaskTracker: Starting
> tasktracker with owner as hadoop
> 2012-12-03 01:28:17,752 INFO org.apache.hadoop.mapred.TaskTracker: Good
> mapred local directories are:
> /mnt/hdfs/data1/mapred,/mnt/hdfs/data2/mapred,/mnt/hdfs/data3/mapred,/mnt/hdfs/data4/mapred,…
>
> It takes 10+ minutes to get past this last line, and then it finally
> continues on and registers in ok with the JT.
>
>
> Thanks

--
Harsh J

Ahh that makes perfect sense.  Thank you!

On 12/2/12 9:25 PM, "Harsh J" <[EMAIL PROTECTED]> wrote:

>Hi,
>
>This is cause of the TT's behavior of deleting the mapred.local.dir
>contents every time you restart it. In your version, 1.0.3, that
>process is synchronous and hence it appears like the TT hangs when
>there's a lot of data to purge out from those dirs.
>
>On Mon, Dec 3, 2012 at 7:11 AM, Sean McNamara
><[EMAIL PROTECTED]> wrote:
>> I have a TaskTracker on a particular node that is very slow to join the
>> jobtracker.  When I start it up with ./hadoop-daemon.sh start
>>tasktracker I
>> see the daemon fire up and running in top.  The TaskTracker daemon will
>>sit
>> there using 50% cpu according to top.   If it helps any this cluster is
>>on
>> hadoop 1.0.3.  Does anyone know what the TT could be up to?
>>
>> Here is the log output:
>>
>> 2012-12-03 01:28:17,310 INFO
>>org.apache.hadoop.metrics2.impl.MetricsConfig:
>> loaded properties from hadoop-metrics2.properties
>> 2012-12-03 01:28:17,320 INFO
>> org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source
>> MetricsSystem,sub=Stats registered.
>> 2012-12-03 01:28:17,321 INFO
>> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot
>>period
>> at 10 second(s).
>> 2012-12-03 01:28:17,321 INFO
>> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: TaskTracker metrics
>> system started
>> 2012-12-03 01:28:17,515 INFO
>> org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source
>>ugi
>> registered.
>> 2012-12-03 01:28:17,679 INFO org.mortbay.log: Logging to
>> org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via
>> org.mortbay.log.Slf4jLog
>> 2012-12-03 01:28:17,726 INFO org.apache.hadoop.http.HttpServer: Added
>>global
>> filtersafety
>>(class=org.apache.hadoop.http.HttpServer$QuotingInputFilter)
>> 2012-12-03 01:28:17,728 INFO org.apache.hadoop.http.HttpServer: Added
>>filter
>> static_user_filter
>> (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter)
>>to
>> context WepAppsContext
>> 2012-12-03 01:28:17,728 INFO org.apache.hadoop.http.HttpServer: Added
>>filter
>> static_user_filter
>> (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter)
>>to
>> context static
>> 2012-12-03 01:28:17,728 INFO org.apache.hadoop.http.HttpServer: Added
>>filter
>> static_user_filter
>> (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter)
>>to
>> context logs
>> 2012-12-03 01:28:17,747 INFO org.apache.hadoop.mapred.TaskLogsTruncater:
>> Initializing logs' truncater with mapRetainSize=-1 and
>>reduceRetainSize=-1
>> 2012-12-03 01:28:17,751 INFO org.apache.hadoop.mapred.TaskTracker:
>>Starting
>> tasktracker with owner as hadoop
>> 2012-12-03 01:28:17,752 INFO org.apache.hadoop.mapred.TaskTracker: Good
>> mapred local directories are:
>>
>>/mnt/hdfs/data1/mapred,/mnt/hdfs/data2/mapred,/mnt/hdfs/data3/mapred,/mnt
>>/hdfs/data4/mapred,Š
>>
>> It takes 10+ minutes to get past this last line, and then it finally
>> continues on and registers in ok with the JT.
>>
>>
>> Thanks
>
>
>
>--
>Harsh J