Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Hive, mail # user - is that possible to use Hive 0.11 with libthrift 0.7


+
Zhang Xiaoyu 2013-10-10, 19:09
+
Edward Capriolo 2013-10-10, 19:19
+
Zhang Xiaoyu 2013-10-10, 19:23
Copy link to this message
-
Re: is that possible to use Hive 0.11 with libthrift 0.7
Timothy Potter 2013-10-10, 19:28
Hi Zhang,

I have the same issue in that I use some Cassandra client API's that depend
on Thrift 0.7 and HCatalog 0.11 that depends on Thrift 0.9. I opted for
using the jarjar utility to "shade" the thrift 0.9 classes. Here's what I
added to the build.xml file for the hcatalog-pig-adapter project:

<target name="jarjar">
    <echo message="basedir=${basedir}"/>
    <copy file="${basedir}/../core/build/hcatalog-core-0.11.0.jar"
todir="${basedir}/deps"/>
    <copy file="${basedir}/build/hcatalog-pig-adapter-0.11.0.jar"
todir="${basedir}/deps"/>
    <copy todir="${basedir}/deps">
      <fileset dir="${basedir}/../../build/dist/lib" includes="hive-*.jar"/>
    </copy>
    <taskdef name="jarjar" classname="com.tonicsystems.jarjar.JarJarTask"
classpath="lib/jarjar-1.4.jar"/>
    <jarjar jarfile="hcat-pig-shaded.jar">
        <zipgroupfileset dir="deps" includes="*.jar"/>
        <rule pattern="org.apache.thrift.**" result="org.shaded.thrift.@1"/>
    </jarjar>
</target>

I basically cooked up a deps directory that contains all the JARs that the
hcatalog-pig-adapter depends on and then used jarjar to create a single jar
with the thrift classes renamed at the bytecode level to org.shaded.thrift

You can do something similar for the JDBC code.

Cheers,
Tim

On Thu, Oct 10, 2013 at 1:19 PM, Edward Capriolo <[EMAIL PROTECTED]>wrote:

> You are kinda screwed. Thrift is wire-compatible in many cases but not API
> compatible. You can not have two applications build off two versions of
> thrift in the same classpath without something like OSGI. To insulate the
> class loaders from each other.
>
> Your best bet is upgrading "other component" to thrift 9
>
>
> On Thu, Oct 10, 2013 at 3:09 PM, Zhang Xiaoyu <[EMAIL PROTECTED]>wrote:
>
>> Hi, all,
>> I am writing a piece of code talking to Hive 0.11 Hive Server 2. The JDBC
>> code depends on libthrift 0.9. However one component which depends on
>> libthrift 0.7 and not binary compatible with libthrift 0.9.
>>
>> When I downgrade to 0.7, I got below NoClassDefFoundError:
>> org/apache/thrift/scheme/StandardScheme.
>>
>> Any idea how to make HS2 working with thrift 0.7? Thanks.
>>
>> Johnny
>>
>
>
+
Zhang Xiaoyu 2013-10-10, 23:31
+
Timothy Potter 2013-10-11, 00:04
+
Timothy Potter 2013-10-11, 00:12