Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hive >> mail # user >> is that possible to use Hive 0.11 with libthrift 0.7


Copy link to this message
-
Re: is that possible to use Hive 0.11 with libthrift 0.7
Hi Zhang,

I have the same issue in that I use some Cassandra client API's that depend
on Thrift 0.7 and HCatalog 0.11 that depends on Thrift 0.9. I opted for
using the jarjar utility to "shade" the thrift 0.9 classes. Here's what I
added to the build.xml file for the hcatalog-pig-adapter project:

<target name="jarjar">
    <echo message="basedir=${basedir}"/>
    <copy file="${basedir}/../core/build/hcatalog-core-0.11.0.jar"
todir="${basedir}/deps"/>
    <copy file="${basedir}/build/hcatalog-pig-adapter-0.11.0.jar"
todir="${basedir}/deps"/>
    <copy todir="${basedir}/deps">
      <fileset dir="${basedir}/../../build/dist/lib" includes="hive-*.jar"/>
    </copy>
    <taskdef name="jarjar" classname="com.tonicsystems.jarjar.JarJarTask"
classpath="lib/jarjar-1.4.jar"/>
    <jarjar jarfile="hcat-pig-shaded.jar">
        <zipgroupfileset dir="deps" includes="*.jar"/>
        <rule pattern="org.apache.thrift.**" result="org.shaded.thrift.@1"/>
    </jarjar>
</target>

I basically cooked up a deps directory that contains all the JARs that the
hcatalog-pig-adapter depends on and then used jarjar to create a single jar
with the thrift classes renamed at the bytecode level to org.shaded.thrift

You can do something similar for the JDBC code.

Cheers,
Tim

On Thu, Oct 10, 2013 at 1:19 PM, Edward Capriolo <[EMAIL PROTECTED]>wrote:

> You are kinda screwed. Thrift is wire-compatible in many cases but not API
> compatible. You can not have two applications build off two versions of
> thrift in the same classpath without something like OSGI. To insulate the
> class loaders from each other.
>
> Your best bet is upgrading "other component" to thrift 9
>
>
> On Thu, Oct 10, 2013 at 3:09 PM, Zhang Xiaoyu <[EMAIL PROTECTED]>wrote:
>
>> Hi, all,
>> I am writing a piece of code talking to Hive 0.11 Hive Server 2. The JDBC
>> code depends on libthrift 0.9. However one component which depends on
>> libthrift 0.7 and not binary compatible with libthrift 0.9.
>>
>> When I downgrade to 0.7, I got below NoClassDefFoundError:
>> org/apache/thrift/scheme/StandardScheme.
>>
>> Any idea how to make HS2 working with thrift 0.7? Thanks.
>>
>> Johnny
>>
>
>
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB