Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Sqoop, mail # user - Import failure due to thrift incompatibility?


+
Lars Yencken 2013-10-08, 06:26
+
Jarek Jarcec Cecho 2013-10-10, 15:54
Copy link to this message
-
Re: Import failure due to thrift incompatibility?
Lars Yencken 2013-10-11, 04:56
Hi Jarek,

I'm not using HBase on that cluster, which I've verified by hunting for
binaries. Sqoop seems to be pulling it in though:

hadoop@ip-XXX-XXX-XXX-XXX:~/sqoop$ find . -iname '*hbase*.jar'
./build/ivy/lib/sqoop/hadoop100test/hbase-0.92.0-tests.jar
./build/ivy/lib/sqoop/hadoop100test/hbase-0.92.0.jar

Any idea if I can prune it from Sqoop's build?

Cheers,
Lars
On 11 October 2013 02:54, Jarek Jarcec Cecho <[EMAIL PROTECTED]> wrote:

> Hi Lars,
> I've seen this issue many times and I can confirm your findings - this
> exception is most likely being thrown because multiple incompatible thrift
> implementations are available on Sqoop classpath.
>
> However Sqoop itself do not ships with thrift as we are not using thrift.
> Usually it's the other dependent projects that are bringing thrift as
> transitive dependency. Are you by any chance also using HBase? HBase is
> known to ship it's own thrift version that is usually not the same as the
> one from Hive.
>
> Jarcec
>
> On Tue, Oct 08, 2013 at 05:26:52PM +1100, Lars Yencken wrote:
> > Hi all,
> >
> > I'm having trouble with what should be a standard Sqoop import job, from
> > MySQL to Hive. On a new cluster, I get this exception when I do my import
> > run:
> >
> > java.lang.NoSuchMethodError: org.apache.thrift.EncodingUtils.setBit(BIZ)B
> >
> > Full stack trace: https://gist.github.com/larsyencken/6879918
> > Environment: Sqoop 1.4.4 / Hive 0.8.1.8 / Hadoop 1.0.3 on EMR
> >
> > The problem only occurs with the Hive import. If I take it out and just
> > import from MySQL to HDFS, the data comes through without problems.
> >
> > I have a feeling the problem is to do with multiple incompatible thrift
> > versions (Hive has libthrift-0.7.0.jar, Sqoop uses libthrift-0.9.0.jar).
> >
> > Any suggestions on getting these two to play nicely together?
> >
> > Best,
> > Lars
>
+
Jarek Jarcec Cecho 2013-10-11, 15:36