Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hadoop, mail # user - Local jobtracker in test env?


Copy link to this message
-
Re: Local jobtracker in test env?
Harsh J 2012-08-07, 21:08
It used the local mode of operation: org.apache.hadoop.mapred.LocalJobRunner

A JobTracker (via MiniMRCluster) is only required for simulating
distributed tests.

On Wed, Aug 8, 2012 at 2:27 AM, Mohit Anchlia <[EMAIL PROTECTED]> wrote:
> I just wrote a test where fs.default.name is file:/// and
> mapred.job.tracker is set to local. The test ran fine, I also see mapper
> and reducer were invoked but what I am trying to understand is that how did
> this run without specifying the job tracker port and which port task
> tracker connected with job tracker. It's not clear from the output:
>
> Also what's the difference between this and bringing up miniDFS cluster?
>
> INFO  org.apache.hadoop.mapred.FileInputFormat [main]: Total input paths to
> proc
> ess : 1
> INFO  org.apache.hadoop.mapred.JobClient [main]: Running job: job_local_0001
> INFO  org.apache.hadoop.mapred.Task [Thread-11]:  Using
> ResourceCalculatorPlugin
>  : null
> INFO  org.apache.hadoop.mapred.MapTask [Thread-11]: numReduceTasks: 1
> INFO  org.apache.hadoop.mapred.MapTask [Thread-11]: io.sort.mb = 100
> INFO  org.apache.hadoop.mapred.MapTask [Thread-11]: data buffer > 79691776/99614
> 720
> INFO  org.apache.hadoop.mapred.MapTask [Thread-11]: record buffer > 262144/32768
> 0
> INFO  com.i.cg.services.dp.analytics.hadoop.mapred.GeoLookup [Thread-11]: z
> ip 92127
> INFO  com.i.cg.services.dp.analytics.hadoop.mapred.GeoLookup [Thread-11]: z
> ip 1
> INFO  com.i.cg.services.dp.analytics.hadoop.mapred.GeoLookup [Thread-11]: z
> ip 92127
> INFO  com.i.cg.services.dp.analytics.hadoop.mapred.GeoLookup [Thread-11]: z
> ip 1
> INFO  org.apache.hadoop.mapred.MapTask [Thread-11]: Starting flush of map
> output
> INFO  org.apache.hadoop.mapred.MapTask [Thread-11]: Finished spill 0
> INFO  org.apache.hadoop.mapred.Task [Thread-11]:
> Task:attempt_local_0001_m_00000
> 0_0 is done. And is in the process of commiting
> INFO  org.apache.hadoop.mapred.LocalJobRunner [Thread-11]:
> file:/c:/upb/dp/manch
> lia-dp/depot/services/data-platform/trunk/analytics/geoinput/geo.dat:0+18
> INFO  org.apache.hadoop.mapred.Task [Thread-11]: Task
> 'attempt_local_0001_m_0000
> 00_0' done.
> INFO  org.apache.hadoop.mapred.Task [Thread-11]:  Using
> ResourceCalculatorPlugin
>  : null
> INFO  org.apache.hadoop.mapred.LocalJobRunner [Thread-11]:
> INFO  org.apache.hadoop.mapred.Merger [Thread-11]: Merging 1 sorted segments
> INFO  org.apache.hadoop.mapred.Merger [Thread-11]: Down to the last
> merge-pass,
> with 1 segments left of total size: 26 bytes
> INFO  org.apache.hadoop.mapred.LocalJobRunner [Thread-11]:
> INFO  com.i.cg.services.dp.analytics.hadoop.mapred.GeoLookup [Thread-11]: I
> nside reduce
> INFO  com.i.cg.services.dp.analytics.hadoop.mapred.GeoLookup [Thread-11]: O
> utside reduce
> INFO  org.apache.hadoop.mapred.Task [Thread-11]:
> Task:attempt_local_0001_r_00000
> 0_0 is done. And is in the process of commiting
> INFO  org.apache.hadoop.mapred.LocalJobRunner [Thread-11]:
> INFO  org.apache.hadoop.mapred.Task [Thread-11]: Task
> attempt_local_0001_r_00000
> 0_0 is allowed to commit now
> INFO  org.apache.hadoop.mapred.FileOutputCommitter [Thread-11]: Saved
> output of
> task 'attempt_local_0001_r_000000_0' to
> file:/c:/upb/dp/manchlia-dp/depot/servic
> es/data-platform/trunk/analytics/geooutput
> INFO  org.apache.hadoop.mapred.LocalJobRunner [Thread-11]: reduce > reduce
> INFO  org.apache.hadoop.mapred.Task [Thread-11]: Task
> 'attempt_local_0001_r_0000
> 00_0' done.
> INFO  org.apache.hadoop.mapred.JobClient [main]:  map 100% reduce 100%
> INFO  org.apache.hadoop.mapred.JobClient [main]: Job complete:
> job_local_0001
> INFO  org.apache.hadoop.mapred.JobClient [main]: Counters: 15
> INFO  org.apache.hadoop.mapred.JobClient [main]:   FileSystemCounters
> INFO  org.apache.hadoop.mapred.JobClient [main]:     FILE_BYTES_READ=458
> INFO  org.apache.hadoop.mapred.JobClient [main]:
> FILE_BYTES_WRITTEN=96110
> INFO  org.apache.hadoop.mapred.JobClient [main]:   Map-Reduce Framework
> INFO  org.apache.hadoop.mapred.JobClient [main]:     Map input records=2

Harsh J