Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
MapReduce >> mail # user >> "Type mismatch in key from map: expected org.apache.hadoop.io.LongWritable" using MultipleInputs (multiple mappers)


Copy link to this message
-
Re: "Type mismatch in key from map: expected org.apache.hadoop.io.LongWritable" using MultipleInputs (multiple mappers)
It looks like you're using the local job runner which does everything in
a single thread. In this case, yes, I think the mappers are run
sequentially. The local job runner is a different code path in Hadoop
and is a known issue. Have you tried your code in pseudo-distributed mode?

HTH.

On 2/11/10 11:14 AM, Martin Häger wrote:
> Hello,
>
> We're trying to do a reduce-side join by applying two different
> mappers (TransformationSessionMapper and TransformationActionMapper)
> to two different input files and joining them using
> TransformationReducer. See attached Classify.java for complete source.
>
> When running it, we get the following error. JoinKey is our own
> implementation that is used for performing secondary sort. Somehow
> TransformationActionMapper gets passed a JoinKey when it expects a
> LongWritable (TextInputFormat). Is Hadoop actually applying the
> mappers in sequence?
>
> $ hadoop jar /tmp/classify.jar Classify
> 10/02/11 16:40:16 INFO jvm.JvmMetrics: Initializing JVM Metrics with
> processName=JobTracker, sessionId> 10/02/11 16:40:16 WARN mapred.JobClient: No job jar file set.  User
> classes may not be found. See JobConf(Class) or
> JobConf#setJar(String).
> 10/02/11 16:40:16 INFO jvm.JvmMetrics: Cannot initialize JVM Metrics
> with processName=JobTracker, sessionId= - already initialized
> 10/02/11 16:40:16 INFO input.FileInputFormat: Total input paths to process : 1
> 10/02/11 16:40:16 INFO input.FileInputFormat: Total input paths to process : 1
> 10/02/11 16:40:16 INFO mapred.JobClient: Running job: job_local_0001
> 10/02/11 16:40:16 INFO jvm.JvmMetrics: Cannot initialize JVM Metrics
> with processName=JobTracker, sessionId= - already initialized
> 10/02/11 16:40:16 INFO input.FileInputFormat: Total input paths to process : 1
> 10/02/11 16:40:16 INFO input.FileInputFormat: Total input paths to process : 1
> 10/02/11 16:40:16 INFO mapred.MapTask: io.sort.mb = 100
> 10/02/11 16:40:16 INFO mapred.MapTask: data buffer = 79691776/99614720
> 10/02/11 16:40:16 INFO mapred.MapTask: record buffer = 262144/327680
> 10/02/11 16:40:16 WARN mapred.LocalJobRunner: job_local_0001
> java.io.IOException: Type mismatch in key from map: expected
> org.apache.hadoop.io.LongWritable, recieved Classify$JoinKey
> at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.collect(MapTask.java:807)
> at org.apache.hadoop.mapred.MapTask$NewOutputCollector.write(MapTask.java:504)
> at org.apache.hadoop.mapreduce.TaskInputOutputContext.write(TaskInputOutputContext.java:80)
> at Classify$TransformationActionMapper.map(Classify.java:161)
> at Classify$TransformationActionMapper.map(Classify.java:1)
> at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
> at org.apache.hadoop.mapreduce.lib.input.DelegatingMapper.run(DelegatingMapper.java:51)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:583)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305)
> at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:176)
> 10/02/11 16:40:17 INFO mapred.JobClient:  map 0% reduce 0%
> 10/02/11 16:40:17 INFO mapred.JobClient: Job complete: job_local_0001
> 10/02/11 16:40:17 INFO mapred.JobClient: Counters: 0
--
Eric Sammer
[EMAIL PROTECTED]
http://esammer.blogspot.com