Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
MapReduce >> mail # user >> "Type mismatch in key from map: expected org.apache.hadoop.io.LongWritable" using MultipleInputs (multiple mappers)


Copy link to this message
-
Re: "Type mismatch in key from map: expected org.apache.hadoop.io.LongWritable" using MultipleInputs (multiple mappers)
It looks like you're using the local job runner which does everything in
a single thread. In this case, yes, I think the mappers are run
sequentially. The local job runner is a different code path in Hadoop
and is a known issue. Have you tried your code in pseudo-distributed mode?

HTH.

On 2/11/10 11:14 AM, Martin Häger wrote:
> Hello,
>
> We're trying to do a reduce-side join by applying two different
> mappers (TransformationSessionMapper and TransformationActionMapper)
> to two different input files and joining them using
> TransformationReducer. See attached Classify.java for complete source.
>
> When running it, we get the following error. JoinKey is our own
> implementation that is used for performing secondary sort. Somehow
> TransformationActionMapper gets passed a JoinKey when it expects a
> LongWritable (TextInputFormat). Is Hadoop actually applying the
> mappers in sequence?
>
> $ hadoop jar /tmp/classify.jar Classify
> 10/02/11 16:40:16 INFO jvm.JvmMetrics: Initializing JVM Metrics with
> processName=JobTracker, sessionId> 10/02/11 16:40:16 WARN mapred.JobClient: No job jar file set.  User
> classes may not be found. See JobConf(Class) or
> JobConf#setJar(String).
> 10/02/11 16:40:16 INFO jvm.JvmMetrics: Cannot initialize JVM Metrics
> with processName=JobTracker, sessionId= - already initialized
> 10/02/11 16:40:16 INFO input.FileInputFormat: Total input paths to process : 1
> 10/02/11 16:40:16 INFO input.FileInputFormat: Total input paths to process : 1
> 10/02/11 16:40:16 INFO mapred.JobClient: Running job: job_local_0001
> 10/02/11 16:40:16 INFO jvm.JvmMetrics: Cannot initialize JVM Metrics
> with processName=JobTracker, sessionId= - already initialized
> 10/02/11 16:40:16 INFO input.FileInputFormat: Total input paths to process : 1
> 10/02/11 16:40:16 INFO input.FileInputFormat: Total input paths to process : 1
> 10/02/11 16:40:16 INFO mapred.MapTask: io.sort.mb = 100
> 10/02/11 16:40:16 INFO mapred.MapTask: data buffer = 79691776/99614720
> 10/02/11 16:40:16 INFO mapred.MapTask: record buffer = 262144/327680
> 10/02/11 16:40:16 WARN mapred.LocalJobRunner: job_local_0001
> java.io.IOException: Type mismatch in key from map: expected
> org.apache.hadoop.io.LongWritable, recieved Classify$JoinKey
> at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.collect(MapTask.java:807)
> at org.apache.hadoop.mapred.MapTask$NewOutputCollector.write(MapTask.java:504)
> at org.apache.hadoop.mapreduce.TaskInputOutputContext.write(TaskInputOutputContext.java:80)
> at Classify$TransformationActionMapper.map(Classify.java:161)
> at Classify$TransformationActionMapper.map(Classify.java:1)
> at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
> at org.apache.hadoop.mapreduce.lib.input.DelegatingMapper.run(DelegatingMapper.java:51)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:583)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305)
> at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:176)
> 10/02/11 16:40:17 INFO mapred.JobClient:  map 0% reduce 0%
> 10/02/11 16:40:17 INFO mapred.JobClient: Job complete: job_local_0001
> 10/02/11 16:40:17 INFO mapred.JobClient: Counters: 0
--
Eric Sammer
[EMAIL PROTECTED]
http://esammer.blogspot.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB