Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
MapReduce, mail # user - Exception comes out with the override option in CompositeInputFormat


Copy link to this message
-
Exception comes out with the override option in CompositeInputFormat
YouPeng Yang 2013-05-19, 08:44
Hi All
  I‘m trying the CompositeInputFormat to perform the Map-side join。 The
outer and inner join go well. However I try the override option .It come
out an exception[1]:

  My JobDriver class as follows:

public static  void main(String[] args) throws IOException,
InterruptedException, ClassNotFoundException{
Configuration conf  = new  Configuration();
conf.set("mapreduce.input.keyvaluelinerecordreader.key.value.separator","|");
Job job = new Job(conf,"this is a  test");

job.setInputFormatClass(CompositeInputFormat.class);
job.getConfiguration().set(CompositeInputFormat.JOIN_EXPR,CompositeInputFormat.compose("
override",KeyValueTextInputFormat.class,
"/user/hadoop/maptest_a.txt",
"/user/hadoop/maptest_b.txt",
"/user/hadoop/maptest_c.txt"));
 job.setOutputFormatClass(TextOutputFormat.class);
job.setOutputKeyClass(Text.class);
job.setOutputValueClass(Text.class);
job.setMapperClass(JoinMappper.class);
FileOutputFormat.setOutputPath(job,new Path("test/jobtest8/"));
job.waitForCompletion(true);
}

Is there something wrong with My code?
and How does the override option impact on the join work?

[1]2013-05-19 16:22:52,201 INFO  mapred.LocalJobRunner
(LocalJobRunner.java:run(394)) - Map task executor complete.
2013-05-19 16:22:52,213 WARN  mapred.LocalJobRunner
(LocalJobRunner.java:run(479)) - job_local_0001
java.lang.Exception: java.lang.ClassCastException:
org.apache.hadoop.io.Text cannot be cast to
org.apache.hadoop.mapreduce.lib.join.TupleWritable
at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:400)
Caused by: java.lang.ClassCastException: org.apache.hadoop.io.Text cannot
be cast to org.apache.hadoop.mapreduce.lib.join.TupleWritable
at com.test.JoinTestMapper$JoinMappper.map(JoinTestMapper.java:1)
at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:725)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:332)
at
org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:232)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
2013-05-19 16:22:52,635 INFO  mapreduce.Job
(Job.java:monitorAndPrintJob(1286)) - Job job_local_0001 running in uber
mode : false
2013-05-19 16:22:52,635 INFO  mapreduce.Job
(Job.java:monitorAndPrintJob(1293)) -  map 0% reduce 0%
2013-05-19 16:22:52,639 INFO  mapreduce.Job
(Job.java:monitorAndPrintJob(1306)) - Job job_local_0001 failed with state
FAILED due to: NA