Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hive, mail # user - is that possible to use Hive 0.11 with libthrift 0.7


Copy link to this message
-
Re: is that possible to use Hive 0.11 with libthrift 0.7
Zhang Xiaoyu 2013-10-10, 23:31
Hi, Timothy,
Thanks for your reply. Seems jarjar is a solution for me. I have a basic
question to follow:

I don't quite understand what is tag <keep> for ?

My understand is
1.
use <include> to grab the maven dependencies I want to re-package,
2.
then use <pattern> and <result> in <rule> to define what class to
re-package and what's the new name for re-packaged binary.
3.
And use <keep> to indicate which class in the project will only use this
re-packaged dependencies, in my case, it is JDBC class. Am I right ?

below pom doesn't work for me, and JDBC class and other class still share
the same thrift 0.9. Thrift 0.7 is omitted.

Thanks a lot for help,
Johnny

<dependency>

<groupId>org.apache.thrift</groupId>

<artifactId>libthrift</artifactId>

<version>0.9.0</version>

</dependency>

......

......

<plugin>

<groupId>org.sonatype.plugins</groupId>

      <artifactId>jarjar-maven-plugin</artifactId>

      <executions>

           <execution>

             <phase>package</phase>

             <goals>

               <goal>jarjar</goal>

             </goals>

             <configuration>

               <includes>

                 <include>*org.apache.thrift:libthrift*</include>

               </includes>

               <rules>

                 <rule>

                   <pattern>org.apache.thrift.**</pattern>

                   <result>com.xxx.JDBC.class.pattern.internal.thrift.@1</
result>

                 </rule>

                 <keep>

                   <pattern>com.xxx.JDBC.class.pattern.**</pattern>

                  </keep>

               </rules>

             </configuration>

           </execution>

        </executions>

</plugin>
On Thu, Oct 10, 2013 at 12:28 PM, Timothy Potter <[EMAIL PROTECTED]>wrote:

> Hi Zhang,
>
> I have the same issue in that I use some Cassandra client API's that
> depend on Thrift 0.7 and HCatalog 0.11 that depends on Thrift 0.9. I opted
> for using the jarjar utility to "shade" the thrift 0.9 classes. Here's what
> I added to the build.xml file for the hcatalog-pig-adapter project:
>
> <target name="jarjar">
>     <echo message="basedir=${basedir}"/>
>     <copy file="${basedir}/../core/build/hcatalog-core-0.11.0.jar"
> todir="${basedir}/deps"/>
>     <copy file="${basedir}/build/hcatalog-pig-adapter-0.11.0.jar"
> todir="${basedir}/deps"/>
>     <copy todir="${basedir}/deps">
>       <fileset dir="${basedir}/../../build/dist/lib"
> includes="hive-*.jar"/>
>     </copy>
>     <taskdef name="jarjar" classname="com.tonicsystems.jarjar.JarJarTask"
> classpath="lib/jarjar-1.4.jar"/>
>     <jarjar jarfile="hcat-pig-shaded.jar">
>         <zipgroupfileset dir="deps" includes="*.jar"/>
>         <rule pattern="org.apache.thrift.**" result="org.shaded.thrift.@1
> "/>
>     </jarjar>
> </target>
>
> I basically cooked up a deps directory that contains all the JARs that the
> hcatalog-pig-adapter depends on and then used jarjar to create a single jar
> with the thrift classes renamed at the bytecode level to org.shaded.thrift
>
> You can do something similar for the JDBC code.
>
> Cheers,
> Tim
>
>
>
> On Thu, Oct 10, 2013 at 1:19 PM, Edward Capriolo <[EMAIL PROTECTED]>wrote:
>
>> You are kinda screwed. Thrift is wire-compatible in many cases but not
>> API compatible. You can not have two applications build off two versions of
>> thrift in the same classpath without something like OSGI. To insulate the
>> class loaders from each other.
>>
>> Your best bet is upgrading "other component" to thrift 9
>>
>>
>> On Thu, Oct 10, 2013 at 3:09 PM, Zhang Xiaoyu <[EMAIL PROTECTED]>wrote:
>>
>>> Hi, all,
>>> I am writing a piece of code talking to Hive 0.11 Hive Server 2. The
>>> JDBC code depends on libthrift 0.9. However one component which depends on
>>> libthrift 0.7 and not binary compatible with libthrift 0.9.
>>>
>>> When I downgrade to 0.7, I got below NoClassDefFoundError:
>>> org/apache/thrift/scheme/StandardScheme.
>>>
>>> Any idea how to make HS2 working with thrift 0.7? Thanks.