Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Hadoop >> mail # user >> Local jobtracker in test env?


Copy link to this message
-
Local jobtracker in test env?
I just wrote a test where fs.default.name is file:/// and
mapred.job.tracker is set to local. The test ran fine, I also see mapper
and reducer were invoked but what I am trying to understand is that how did
this run without specifying the job tracker port and which port task
tracker connected with job tracker. It's not clear from the output:

Also what's the difference between this and bringing up miniDFS cluster?

INFO  org.apache.hadoop.mapred.FileInputFormat [main]: Total input paths to
proc
ess : 1
INFO  org.apache.hadoop.mapred.JobClient [main]: Running job: job_local_0001
INFO  org.apache.hadoop.mapred.Task [Thread-11]:  Using
ResourceCalculatorPlugin
 : null
INFO  org.apache.hadoop.mapred.MapTask [Thread-11]: numReduceTasks: 1
INFO  org.apache.hadoop.mapred.MapTask [Thread-11]: io.sort.mb = 100
INFO  org.apache.hadoop.mapred.MapTask [Thread-11]: data buffer 79691776/99614
720
INFO  org.apache.hadoop.mapred.MapTask [Thread-11]: record buffer 262144/32768
0
INFO  com.i.cg.services.dp.analytics.hadoop.mapred.GeoLookup [Thread-11]: z
ip 92127
INFO  com.i.cg.services.dp.analytics.hadoop.mapred.GeoLookup [Thread-11]: z
ip 1
INFO  com.i.cg.services.dp.analytics.hadoop.mapred.GeoLookup [Thread-11]: z
ip 92127
INFO  com.i.cg.services.dp.analytics.hadoop.mapred.GeoLookup [Thread-11]: z
ip 1
INFO  org.apache.hadoop.mapred.MapTask [Thread-11]: Starting flush of map
output
INFO  org.apache.hadoop.mapred.MapTask [Thread-11]: Finished spill 0
INFO  org.apache.hadoop.mapred.Task [Thread-11]:
Task:attempt_local_0001_m_00000
0_0 is done. And is in the process of commiting
INFO  org.apache.hadoop.mapred.LocalJobRunner [Thread-11]:
file:/c:/upb/dp/manch
lia-dp/depot/services/data-platform/trunk/analytics/geoinput/geo.dat:0+18
INFO  org.apache.hadoop.mapred.Task [Thread-11]: Task
'attempt_local_0001_m_0000
00_0' done.
INFO  org.apache.hadoop.mapred.Task [Thread-11]:  Using
ResourceCalculatorPlugin
 : null
INFO  org.apache.hadoop.mapred.LocalJobRunner [Thread-11]:
INFO  org.apache.hadoop.mapred.Merger [Thread-11]: Merging 1 sorted segments
INFO  org.apache.hadoop.mapred.Merger [Thread-11]: Down to the last
merge-pass,
with 1 segments left of total size: 26 bytes
INFO  org.apache.hadoop.mapred.LocalJobRunner [Thread-11]:
INFO  com.i.cg.services.dp.analytics.hadoop.mapred.GeoLookup [Thread-11]: I
nside reduce
INFO  com.i.cg.services.dp.analytics.hadoop.mapred.GeoLookup [Thread-11]: O
utside reduce
INFO  org.apache.hadoop.mapred.Task [Thread-11]:
Task:attempt_local_0001_r_00000
0_0 is done. And is in the process of commiting
INFO  org.apache.hadoop.mapred.LocalJobRunner [Thread-11]:
INFO  org.apache.hadoop.mapred.Task [Thread-11]: Task
attempt_local_0001_r_00000
0_0 is allowed to commit now
INFO  org.apache.hadoop.mapred.FileOutputCommitter [Thread-11]: Saved
output of
task 'attempt_local_0001_r_000000_0' to
file:/c:/upb/dp/manchlia-dp/depot/servic
es/data-platform/trunk/analytics/geooutput
INFO  org.apache.hadoop.mapred.LocalJobRunner [Thread-11]: reduce > reduce
INFO  org.apache.hadoop.mapred.Task [Thread-11]: Task
'attempt_local_0001_r_0000
00_0' done.
INFO  org.apache.hadoop.mapred.JobClient [main]:  map 100% reduce 100%
INFO  org.apache.hadoop.mapred.JobClient [main]: Job complete:
job_local_0001
INFO  org.apache.hadoop.mapred.JobClient [main]: Counters: 15
INFO  org.apache.hadoop.mapred.JobClient [main]:   FileSystemCounters
INFO  org.apache.hadoop.mapred.JobClient [main]:     FILE_BYTES_READ=458
INFO  org.apache.hadoop.mapred.JobClient [main]:
FILE_BYTES_WRITTEN=96110
INFO  org.apache.hadoop.mapred.JobClient [main]:   Map-Reduce Framework
INFO  org.apache.hadoop.mapred.JobClient [main]:     Map input records=2
INFO  org.apache.hadoop.mapred.JobClient [main]:     Reduce shuffle bytes=0
INFO  org.apache.hadoop.mapred.JobClient [main]:     Spilled Records=4
INFO  org.apache.hadoop.mapred.JobClient [main]:     Map output bytes=20
INFO  org.apache.hadoop.mapred.JobClient [main]:     Total committed heap
usage
(bytes)=321527808
INFO  org.apache.hadoop.mapred.JobClient [main]:     Map input bytes=18
INFO  org.apache.hadoop.mapred.JobClient [main]:     SPLIT_RAW_BYTES=142
INFO  org.apache.hadoop.mapred.JobClient [main]:     Combine input records=0
INFO  org.apache.hadoop.mapred.JobClient [main]:     Reduce input records=2
INFO  org.apache.hadoop.mapred.JobClient [main]:     Reduce input groups=1
INFO  org.apache.hadoop.mapred.JobClient [main]:     Combine output
records=0
INFO  org.apache.hadoop.mapred.JobClient [main]:     Reduce output records=1
INFO  org.apache.hadoop.mapred.JobClient [main]:     Map output records=2
INFO  com.i.cg.services.dp.analytics.hadoop.mapred.GeoLookup [main]: Inside
 reduce
INFO  com.i.cg.services.dp.analytics.hadoop.mapred.GeoLookup [main]: Outsid
e reduce
Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 4.547 sec
Results :
Tests run: 4, Failures: 0, Errors: 0, Skipped: 0
+
Harsh J 2012-08-07, 21:08
+
Mohit Anchlia 2012-08-07, 23:20
+
Harsh J 2012-08-08, 06:21
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB