Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Hadoop >> mail # user >> Local jobtracker in test env?


+
Mohit Anchlia 2012-08-07, 20:57
+
Harsh J 2012-08-07, 21:08
+
Mohit Anchlia 2012-08-07, 23:20
Copy link to this message
-
Re: Local jobtracker in test env?
Yes, singular JVM (The test JVM itself) and the latter approach (no
TT/JT daemons).

On Wed, Aug 8, 2012 at 4:50 AM, Mohit Anchlia <[EMAIL PROTECTED]> wrote:
> On Tue, Aug 7, 2012 at 2:08 PM, Harsh J <[EMAIL PROTECTED]> wrote:
>
>> It used the local mode of operation:
>> org.apache.hadoop.mapred.LocalJobRunner
>>
>>
> In localmode everything is done inside the same JVM i.e.
> tasktracker,jobtracker etc. all run in the same JVM. Or does it mean that
> none of those processes run everything is pipelined in the same process on
> the local file system.
>
>
>> A JobTracker (via MiniMRCluster) is only required for simulating
>> distributed tests.
>>
>> On Wed, Aug 8, 2012 at 2:27 AM, Mohit Anchlia <[EMAIL PROTECTED]>
>> wrote:
>> > I just wrote a test where fs.default.name is file:/// and
>> > mapred.job.tracker is set to local. The test ran fine, I also see mapper
>> > and reducer were invoked but what I am trying to understand is that how
>> did
>> > this run without specifying the job tracker port and which port task
>> > tracker connected with job tracker. It's not clear from the output:
>> >
>> > Also what's the difference between this and bringing up miniDFS cluster?
>> >
>> > INFO  org.apache.hadoop.mapred.FileInputFormat [main]: Total input paths
>> to
>> > proc
>> > ess : 1
>> > INFO  org.apache.hadoop.mapred.JobClient [main]: Running job:
>> job_local_0001
>> > INFO  org.apache.hadoop.mapred.Task [Thread-11]:  Using
>> > ResourceCalculatorPlugin
>> >  : null
>> > INFO  org.apache.hadoop.mapred.MapTask [Thread-11]: numReduceTasks: 1
>> > INFO  org.apache.hadoop.mapred.MapTask [Thread-11]: io.sort.mb = 100
>> > INFO  org.apache.hadoop.mapred.MapTask [Thread-11]: data buffer >> > 79691776/99614
>> > 720
>> > INFO  org.apache.hadoop.mapred.MapTask [Thread-11]: record buffer >> > 262144/32768
>> > 0
>> > INFO  com.i.cg.services.dp.analytics.hadoop.mapred.GeoLookup
>> [Thread-11]: z
>> > ip 92127
>> > INFO  com.i.cg.services.dp.analytics.hadoop.mapred.GeoLookup
>> [Thread-11]: z
>> > ip 1
>> > INFO  com.i.cg.services.dp.analytics.hadoop.mapred.GeoLookup
>> [Thread-11]: z
>> > ip 92127
>> > INFO  com.i.cg.services.dp.analytics.hadoop.mapred.GeoLookup
>> [Thread-11]: z
>> > ip 1
>> > INFO  org.apache.hadoop.mapred.MapTask [Thread-11]: Starting flush of map
>> > output
>> > INFO  org.apache.hadoop.mapred.MapTask [Thread-11]: Finished spill 0
>> > INFO  org.apache.hadoop.mapred.Task [Thread-11]:
>> > Task:attempt_local_0001_m_00000
>> > 0_0 is done. And is in the process of commiting
>> > INFO  org.apache.hadoop.mapred.LocalJobRunner [Thread-11]:
>> > file:/c:/upb/dp/manch
>> > lia-dp/depot/services/data-platform/trunk/analytics/geoinput/geo.dat:0+18
>> > INFO  org.apache.hadoop.mapred.Task [Thread-11]: Task
>> > 'attempt_local_0001_m_0000
>> > 00_0' done.
>> > INFO  org.apache.hadoop.mapred.Task [Thread-11]:  Using
>> > ResourceCalculatorPlugin
>> >  : null
>> > INFO  org.apache.hadoop.mapred.LocalJobRunner [Thread-11]:
>> > INFO  org.apache.hadoop.mapred.Merger [Thread-11]: Merging 1 sorted
>> segments
>> > INFO  org.apache.hadoop.mapred.Merger [Thread-11]: Down to the last
>> > merge-pass,
>> > with 1 segments left of total size: 26 bytes
>> > INFO  org.apache.hadoop.mapred.LocalJobRunner [Thread-11]:
>> > INFO  com.i.cg.services.dp.analytics.hadoop.mapred.GeoLookup
>> [Thread-11]: I
>> > nside reduce
>> > INFO  com.i.cg.services.dp.analytics.hadoop.mapred.GeoLookup
>> [Thread-11]: O
>> > utside reduce
>> > INFO  org.apache.hadoop.mapred.Task [Thread-11]:
>> > Task:attempt_local_0001_r_00000
>> > 0_0 is done. And is in the process of commiting
>> > INFO  org.apache.hadoop.mapred.LocalJobRunner [Thread-11]:
>> > INFO  org.apache.hadoop.mapred.Task [Thread-11]: Task
>> > attempt_local_0001_r_00000
>> > 0_0 is allowed to commit now
>> > INFO  org.apache.hadoop.mapred.FileOutputCommitter [Thread-11]: Saved
>> > output of
>> > task 'attempt_local_0001_r_000000_0' to
>> > file:/c:/upb/dp/manchlia-dp/depot/servic
>> > es/data-platform/trunk/analytics/geooutput

Harsh J
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB