Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
MapReduce >> mail # user >> Exception comes out with the override option in CompositeInputFormat


Copy link to this message
-
Exception comes out with the override option in CompositeInputFormat
Hi All
  I‘m trying the CompositeInputFormat to perform the Map-side join。 The
outer and inner join go well. However I try the override option .It come
out an exception[1]:

  My JobDriver class as follows:

public static  void main(String[] args) throws IOException,
InterruptedException, ClassNotFoundException{
Configuration conf  = new  Configuration();
conf.set("mapreduce.input.keyvaluelinerecordreader.key.value.separator","|");
Job job = new Job(conf,"this is a  test");

job.setInputFormatClass(CompositeInputFormat.class);
job.getConfiguration().set(CompositeInputFormat.JOIN_EXPR,CompositeInputFormat.compose("
override",KeyValueTextInputFormat.class,
"/user/hadoop/maptest_a.txt",
"/user/hadoop/maptest_b.txt",
"/user/hadoop/maptest_c.txt"));
 job.setOutputFormatClass(TextOutputFormat.class);
job.setOutputKeyClass(Text.class);
job.setOutputValueClass(Text.class);
job.setMapperClass(JoinMappper.class);
FileOutputFormat.setOutputPath(job,new Path("test/jobtest8/"));
job.waitForCompletion(true);
}

Is there something wrong with My code?
and How does the override option impact on the join work?

[1]2013-05-19 16:22:52,201 INFO  mapred.LocalJobRunner
(LocalJobRunner.java:run(394)) - Map task executor complete.
2013-05-19 16:22:52,213 WARN  mapred.LocalJobRunner
(LocalJobRunner.java:run(479)) - job_local_0001
java.lang.Exception: java.lang.ClassCastException:
org.apache.hadoop.io.Text cannot be cast to
org.apache.hadoop.mapreduce.lib.join.TupleWritable
at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:400)
Caused by: java.lang.ClassCastException: org.apache.hadoop.io.Text cannot
be cast to org.apache.hadoop.mapreduce.lib.join.TupleWritable
at com.test.JoinTestMapper$JoinMappper.map(JoinTestMapper.java:1)
at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:725)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:332)
at
org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:232)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
2013-05-19 16:22:52,635 INFO  mapreduce.Job
(Job.java:monitorAndPrintJob(1286)) - Job job_local_0001 running in uber
mode : false
2013-05-19 16:22:52,635 INFO  mapreduce.Job
(Job.java:monitorAndPrintJob(1293)) -  map 0% reduce 0%
2013-05-19 16:22:52,639 INFO  mapreduce.Job
(Job.java:monitorAndPrintJob(1306)) - Job job_local_0001 failed with state
FAILED due to: NA
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB