Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
MapReduce >> mail # user >> Re: TaskTracker slow to start/join


+
Harsh J 2012-12-03, 04:25
Copy link to this message
-
Re: TaskTracker slow to start/join
Ahh that makes perfect sense.  Thank you!

On 12/2/12 9:25 PM, "Harsh J" <[EMAIL PROTECTED]> wrote:

>Hi,
>
>This is cause of the TT's behavior of deleting the mapred.local.dir
>contents every time you restart it. In your version, 1.0.3, that
>process is synchronous and hence it appears like the TT hangs when
>there's a lot of data to purge out from those dirs.
>
>On Mon, Dec 3, 2012 at 7:11 AM, Sean McNamara
><[EMAIL PROTECTED]> wrote:
>> I have a TaskTracker on a particular node that is very slow to join the
>> jobtracker.  When I start it up with ./hadoop-daemon.sh start
>>tasktracker I
>> see the daemon fire up and running in top.  The TaskTracker daemon will
>>sit
>> there using 50% cpu according to top.   If it helps any this cluster is
>>on
>> hadoop 1.0.3.  Does anyone know what the TT could be up to?
>>
>> Here is the log output:
>>
>> 2012-12-03 01:28:17,310 INFO
>>org.apache.hadoop.metrics2.impl.MetricsConfig:
>> loaded properties from hadoop-metrics2.properties
>> 2012-12-03 01:28:17,320 INFO
>> org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source
>> MetricsSystem,sub=Stats registered.
>> 2012-12-03 01:28:17,321 INFO
>> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot
>>period
>> at 10 second(s).
>> 2012-12-03 01:28:17,321 INFO
>> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: TaskTracker metrics
>> system started
>> 2012-12-03 01:28:17,515 INFO
>> org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source
>>ugi
>> registered.
>> 2012-12-03 01:28:17,679 INFO org.mortbay.log: Logging to
>> org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via
>> org.mortbay.log.Slf4jLog
>> 2012-12-03 01:28:17,726 INFO org.apache.hadoop.http.HttpServer: Added
>>global
>> filtersafety
>>(class=org.apache.hadoop.http.HttpServer$QuotingInputFilter)
>> 2012-12-03 01:28:17,728 INFO org.apache.hadoop.http.HttpServer: Added
>>filter
>> static_user_filter
>> (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter)
>>to
>> context WepAppsContext
>> 2012-12-03 01:28:17,728 INFO org.apache.hadoop.http.HttpServer: Added
>>filter
>> static_user_filter
>> (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter)
>>to
>> context static
>> 2012-12-03 01:28:17,728 INFO org.apache.hadoop.http.HttpServer: Added
>>filter
>> static_user_filter
>> (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter)
>>to
>> context logs
>> 2012-12-03 01:28:17,747 INFO org.apache.hadoop.mapred.TaskLogsTruncater:
>> Initializing logs' truncater with mapRetainSize=-1 and
>>reduceRetainSize=-1
>> 2012-12-03 01:28:17,751 INFO org.apache.hadoop.mapred.TaskTracker:
>>Starting
>> tasktracker with owner as hadoop
>> 2012-12-03 01:28:17,752 INFO org.apache.hadoop.mapred.TaskTracker: Good
>> mapred local directories are:
>>
>>/mnt/hdfs/data1/mapred,/mnt/hdfs/data2/mapred,/mnt/hdfs/data3/mapred,/mnt
>>/hdfs/data4/mapred,Š
>>
>> It takes 10+ minutes to get past this last line, and then it finally
>> continues on and registers in ok with the JT.
>>
>>
>> Thanks
>
>
>
>--
>Harsh J
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB