Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Hadoop, mail # user - Local jobtracker in test env?


+
Mohit Anchlia 2012-08-07, 20:57
+
Harsh J 2012-08-07, 21:08
+
Mohit Anchlia 2012-08-07, 23:20
Copy link to this message
-
Re: Local jobtracker in test env?
Harsh J 2012-08-08, 06:21
Yes, singular JVM (The test JVM itself) and the latter approach (no
TT/JT daemons).

On Wed, Aug 8, 2012 at 4:50 AM, Mohit Anchlia <[EMAIL PROTECTED]> wrote:
> On Tue, Aug 7, 2012 at 2:08 PM, Harsh J <[EMAIL PROTECTED]> wrote:
>
>> It used the local mode of operation:
>> org.apache.hadoop.mapred.LocalJobRunner
>>
>>
> In localmode everything is done inside the same JVM i.e.
> tasktracker,jobtracker etc. all run in the same JVM. Or does it mean that
> none of those processes run everything is pipelined in the same process on
> the local file system.
>
>
>> A JobTracker (via MiniMRCluster) is only required for simulating
>> distributed tests.
>>
>> On Wed, Aug 8, 2012 at 2:27 AM, Mohit Anchlia <[EMAIL PROTECTED]>
>> wrote:
>> > I just wrote a test where fs.default.name is file:/// and
>> > mapred.job.tracker is set to local. The test ran fine, I also see mapper
>> > and reducer were invoked but what I am trying to understand is that how
>> did
>> > this run without specifying the job tracker port and which port task
>> > tracker connected with job tracker. It's not clear from the output:
>> >
>> > Also what's the difference between this and bringing up miniDFS cluster?
>> >
>> > INFO  org.apache.hadoop.mapred.FileInputFormat [main]: Total input paths
>> to
>> > proc
>> > ess : 1
>> > INFO  org.apache.hadoop.mapred.JobClient [main]: Running job:
>> job_local_0001
>> > INFO  org.apache.hadoop.mapred.Task [Thread-11]:  Using
>> > ResourceCalculatorPlugin
>> >  : null
>> > INFO  org.apache.hadoop.mapred.MapTask [Thread-11]: numReduceTasks: 1
>> > INFO  org.apache.hadoop.mapred.MapTask [Thread-11]: io.sort.mb = 100
>> > INFO  org.apache.hadoop.mapred.MapTask [Thread-11]: data buffer >> > 79691776/99614
>> > 720
>> > INFO  org.apache.hadoop.mapred.MapTask [Thread-11]: record buffer >> > 262144/32768
>> > 0
>> > INFO  com.i.cg.services.dp.analytics.hadoop.mapred.GeoLookup
>> [Thread-11]: z
>> > ip 92127
>> > INFO  com.i.cg.services.dp.analytics.hadoop.mapred.GeoLookup
>> [Thread-11]: z
>> > ip 1
>> > INFO  com.i.cg.services.dp.analytics.hadoop.mapred.GeoLookup
>> [Thread-11]: z
>> > ip 92127
>> > INFO  com.i.cg.services.dp.analytics.hadoop.mapred.GeoLookup
>> [Thread-11]: z
>> > ip 1
>> > INFO  org.apache.hadoop.mapred.MapTask [Thread-11]: Starting flush of map
>> > output
>> > INFO  org.apache.hadoop.mapred.MapTask [Thread-11]: Finished spill 0
>> > INFO  org.apache.hadoop.mapred.Task [Thread-11]:
>> > Task:attempt_local_0001_m_00000
>> > 0_0 is done. And is in the process of commiting
>> > INFO  org.apache.hadoop.mapred.LocalJobRunner [Thread-11]:
>> > file:/c:/upb/dp/manch
>> > lia-dp/depot/services/data-platform/trunk/analytics/geoinput/geo.dat:0+18
>> > INFO  org.apache.hadoop.mapred.Task [Thread-11]: Task
>> > 'attempt_local_0001_m_0000
>> > 00_0' done.
>> > INFO  org.apache.hadoop.mapred.Task [Thread-11]:  Using
>> > ResourceCalculatorPlugin
>> >  : null
>> > INFO  org.apache.hadoop.mapred.LocalJobRunner [Thread-11]:
>> > INFO  org.apache.hadoop.mapred.Merger [Thread-11]: Merging 1 sorted
>> segments
>> > INFO  org.apache.hadoop.mapred.Merger [Thread-11]: Down to the last
>> > merge-pass,
>> > with 1 segments left of total size: 26 bytes
>> > INFO  org.apache.hadoop.mapred.LocalJobRunner [Thread-11]:
>> > INFO  com.i.cg.services.dp.analytics.hadoop.mapred.GeoLookup
>> [Thread-11]: I
>> > nside reduce
>> > INFO  com.i.cg.services.dp.analytics.hadoop.mapred.GeoLookup
>> [Thread-11]: O
>> > utside reduce
>> > INFO  org.apache.hadoop.mapred.Task [Thread-11]:
>> > Task:attempt_local_0001_r_00000
>> > 0_0 is done. And is in the process of commiting
>> > INFO  org.apache.hadoop.mapred.LocalJobRunner [Thread-11]:
>> > INFO  org.apache.hadoop.mapred.Task [Thread-11]: Task
>> > attempt_local_0001_r_00000
>> > 0_0 is allowed to commit now
>> > INFO  org.apache.hadoop.mapred.FileOutputCommitter [Thread-11]: Saved
>> > output of
>> > task 'attempt_local_0001_r_000000_0' to
>> > file:/c:/upb/dp/manchlia-dp/depot/servic
>> > es/data-platform/trunk/analytics/geooutput

Harsh J