-Re: Import failure due to thrift incompatibility?
Lars Yencken 2013-10-11, 04:56
I'm not using HBase on that cluster, which I've verified by hunting for
binaries. Sqoop seems to be pulling it in though:
hadoop@ip-XXX-XXX-XXX-XXX:~/sqoop$ find . -iname '*hbase*.jar'
Any idea if I can prune it from Sqoop's build?
On 11 October 2013 02:54, Jarek Jarcec Cecho <[EMAIL PROTECTED]> wrote:
> Hi Lars,
> I've seen this issue many times and I can confirm your findings - this
> exception is most likely being thrown because multiple incompatible thrift
> implementations are available on Sqoop classpath.
> However Sqoop itself do not ships with thrift as we are not using thrift.
> Usually it's the other dependent projects that are bringing thrift as
> transitive dependency. Are you by any chance also using HBase? HBase is
> known to ship it's own thrift version that is usually not the same as the
> one from Hive.
> On Tue, Oct 08, 2013 at 05:26:52PM +1100, Lars Yencken wrote:
> > Hi all,
> > I'm having trouble with what should be a standard Sqoop import job, from
> > MySQL to Hive. On a new cluster, I get this exception when I do my import
> > run:
> > java.lang.NoSuchMethodError: org.apache.thrift.EncodingUtils.setBit(BIZ)B
> > Full stack trace: https://gist.github.com/larsyencken/6879918
> > Environment: Sqoop 1.4.4 / Hive 0.8.1.8 / Hadoop 1.0.3 on EMR
> > The problem only occurs with the Hive import. If I take it out and just
> > import from MySQL to HDFS, the data comes through without problems.
> > I have a feeling the problem is to do with multiple incompatible thrift
> > versions (Hive has libthrift-0.7.0.jar, Sqoop uses libthrift-0.9.0.jar).
> > Any suggestions on getting these two to play nicely together?
> > Best,
> > Lars