Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Sqoop >> mail # user >> Import failure due to thrift incompatibility?


Copy link to this message
-
Re: Import failure due to thrift incompatibility?
Hi lars,
those packages are in build directory that is not part of the distribution tarball. In case that you are compiling Sqoop yourself, I would suggest to call target "ant tar" and distribute the generated tarball from build directory to your cluster rather than copying entire working directory.

Jarcec

On Fri, Oct 11, 2013 at 03:56:45PM +1100, Lars Yencken wrote:
> Hi Jarek,
>
> I'm not using HBase on that cluster, which I've verified by hunting for
> binaries. Sqoop seems to be pulling it in though:
>
> hadoop@ip-XXX-XXX-XXX-XXX:~/sqoop$ find . -iname '*hbase*.jar'
> ./build/ivy/lib/sqoop/hadoop100test/hbase-0.92.0-tests.jar
> ./build/ivy/lib/sqoop/hadoop100test/hbase-0.92.0.jar
>
> Any idea if I can prune it from Sqoop's build?
>
> Cheers,
> Lars
>
>
> On 11 October 2013 02:54, Jarek Jarcec Cecho <[EMAIL PROTECTED]> wrote:
>
> > Hi Lars,
> > I've seen this issue many times and I can confirm your findings - this
> > exception is most likely being thrown because multiple incompatible thrift
> > implementations are available on Sqoop classpath.
> >
> > However Sqoop itself do not ships with thrift as we are not using thrift.
> > Usually it's the other dependent projects that are bringing thrift as
> > transitive dependency. Are you by any chance also using HBase? HBase is
> > known to ship it's own thrift version that is usually not the same as the
> > one from Hive.
> >
> > Jarcec
> >
> > On Tue, Oct 08, 2013 at 05:26:52PM +1100, Lars Yencken wrote:
> > > Hi all,
> > >
> > > I'm having trouble with what should be a standard Sqoop import job, from
> > > MySQL to Hive. On a new cluster, I get this exception when I do my import
> > > run:
> > >
> > > java.lang.NoSuchMethodError: org.apache.thrift.EncodingUtils.setBit(BIZ)B
> > >
> > > Full stack trace: https://gist.github.com/larsyencken/6879918
> > > Environment: Sqoop 1.4.4 / Hive 0.8.1.8 / Hadoop 1.0.3 on EMR
> > >
> > > The problem only occurs with the Hive import. If I take it out and just
> > > import from MySQL to HDFS, the data comes through without problems.
> > >
> > > I have a feeling the problem is to do with multiple incompatible thrift
> > > versions (Hive has libthrift-0.7.0.jar, Sqoop uses libthrift-0.9.0.jar).
> > >
> > > Any suggestions on getting these two to play nicely together?
> > >
> > > Best,
> > > Lars
> >
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB