Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Sqoop >> mail # user >> sqoop generate java class in old mapreduce API?


Copy link to this message
-
Re: 答复 : sqoop generate java class in old mapreduce API?
Hi sir,
Sqoop generated class implements Hadoop interface called Writable, but do not implements interface WritableComparable. Whereas implementing Writable interface is sufficient for serialization it's not sufficient for keys that are serialized from mappers. Instead interface WritableComparable is needed in such case. My guess is that the generated class is used in a mapper in a job with reduce phase that is violating this requirement. Would you mind sharing a small code example that reproduces the issue? (e.g. multiple java classes that together put whole mapreduce job).

Jarcec

On Mon, Aug 12, 2013 at 09:23:20PM +0800, Wu YongBo wrote:
> I used classes generated by sqoop as Key-Value, and set OutputFormat to
> SequenceFileOutputFormat, and the error occurred.
> However, when I simply changed the Key-Value to Text, and OutputFormat to
> TextOutputFormat, it worked!
>
>     SqoopGenClass value = ....;
>
>     context.write(value, NullWritable.get());
> =====>>>>
> context.write(new Text(value.toString()), NullWritable.get());
>
> Our hadoop version is 0.20.2,that's too old so I gusss this is the cause.
>
> -----邮件原件-----
> 发件人: Jarek Jarcec Cecho [mailto:[EMAIL PROTECTED]]
> 发送时间: 12 August 2013 08:35
> 收件人: [EMAIL PROTECTED]
> 主题: Re: sqoop generate java class in old mapreduce API?
>
> Hi Yongbo,
> I believe that the writable itself should be independent on the new vs. old
> mapreduce API. Would you mind sharing more details about your use case?
> Maybe a small code example reproducing the issue could help us move forward.
>
> Jarcec
>
> On Thu, Aug 08, 2013 at 09:10:50PM +0800, Wu YongBo wrote:
> > Hi All,
> >
> >  
> >
> >   I am using a java class generated by sqoop(1.4.2),  and I get the
> > following error:
> >
> >  
> >
> > 2013-08-08 20:43:10,854 WARN org.apache.hadoop.mapred.TaskTracker:
> > Error running child
> >
> > java.lang.ClassCastException: class
> > com.a.semv.io.semreportbase_keyword
> >
> >         at java.lang.Class.asSubclass(Class.java:3018)
> >
> >         at
> > org.apache.hadoop.mapred.JobConf.getOutputKeyComparator(JobConf.java:5
> > 99)
> >
> >         at
> > org.apache.hadoop.mapred.MapTask$MapOutputBuffer.<init>(MapTask.java:7
> > 91)
> >
> >         at
> > org.apache.hadoop.mapred.MapTask$NewOutputCollector.<init>(MapTask.jav
> > a:524)
> >
> >         at
> > org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:613)
> >
> >         at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305)
> >
> >         at org.apache.hadoop.mapred.Child.main(Child.java:170)
> >
> >  
> >
> > My job is written in mapreduce new API.
> >
> > My hadoop version:0.20.2
> >
> >  
> >
> > Thanks,
> >
> > Yongbo
> >
> >  
> >
>