Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
MapReduce, mail # user - Re: TaskTracker slow to start/join


Copy link to this message
-
Re: TaskTracker slow to start/join
Sean McNamara 2012-12-03, 04:27
Ahh that makes perfect sense.  Thank you!

On 12/2/12 9:25 PM, "Harsh J" <[EMAIL PROTECTED]> wrote:

>Hi,
>
>This is cause of the TT's behavior of deleting the mapred.local.dir
>contents every time you restart it. In your version, 1.0.3, that
>process is synchronous and hence it appears like the TT hangs when
>there's a lot of data to purge out from those dirs.
>
>On Mon, Dec 3, 2012 at 7:11 AM, Sean McNamara
><[EMAIL PROTECTED]> wrote:
>> I have a TaskTracker on a particular node that is very slow to join the
>> jobtracker.  When I start it up with ./hadoop-daemon.sh start
>>tasktracker I
>> see the daemon fire up and running in top.  The TaskTracker daemon will
>>sit
>> there using 50% cpu according to top.   If it helps any this cluster is
>>on
>> hadoop 1.0.3.  Does anyone know what the TT could be up to?
>>
>> Here is the log output:
>>
>> 2012-12-03 01:28:17,310 INFO
>>org.apache.hadoop.metrics2.impl.MetricsConfig:
>> loaded properties from hadoop-metrics2.properties
>> 2012-12-03 01:28:17,320 INFO
>> org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source
>> MetricsSystem,sub=Stats registered.
>> 2012-12-03 01:28:17,321 INFO
>> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot
>>period
>> at 10 second(s).
>> 2012-12-03 01:28:17,321 INFO
>> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: TaskTracker metrics
>> system started
>> 2012-12-03 01:28:17,515 INFO
>> org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source
>>ugi
>> registered.
>> 2012-12-03 01:28:17,679 INFO org.mortbay.log: Logging to
>> org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via
>> org.mortbay.log.Slf4jLog
>> 2012-12-03 01:28:17,726 INFO org.apache.hadoop.http.HttpServer: Added
>>global
>> filtersafety
>>(class=org.apache.hadoop.http.HttpServer$QuotingInputFilter)
>> 2012-12-03 01:28:17,728 INFO org.apache.hadoop.http.HttpServer: Added
>>filter
>> static_user_filter
>> (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter)
>>to
>> context WepAppsContext
>> 2012-12-03 01:28:17,728 INFO org.apache.hadoop.http.HttpServer: Added
>>filter
>> static_user_filter
>> (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter)
>>to
>> context static
>> 2012-12-03 01:28:17,728 INFO org.apache.hadoop.http.HttpServer: Added
>>filter
>> static_user_filter
>> (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter)
>>to
>> context logs
>> 2012-12-03 01:28:17,747 INFO org.apache.hadoop.mapred.TaskLogsTruncater:
>> Initializing logs' truncater with mapRetainSize=-1 and
>>reduceRetainSize=-1
>> 2012-12-03 01:28:17,751 INFO org.apache.hadoop.mapred.TaskTracker:
>>Starting
>> tasktracker with owner as hadoop
>> 2012-12-03 01:28:17,752 INFO org.apache.hadoop.mapred.TaskTracker: Good
>> mapred local directories are:
>>
>>/mnt/hdfs/data1/mapred,/mnt/hdfs/data2/mapred,/mnt/hdfs/data3/mapred,/mnt
>>/hdfs/data4/mapred,Š
>>
>> It takes 10+ minutes to get past this last line, and then it finally
>> continues on and registers in ok with the JT.
>>
>>
>> Thanks
>
>
>
>--
>Harsh J