Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hive, mail # user - is that possible to use Hive 0.11 with libthrift 0.7


Copy link to this message
-
Re: is that possible to use Hive 0.11 with libthrift 0.7
Timothy Potter 2013-10-11, 00:04
I'm not sure about keep, but if you're going to use maven, go with the
shade plugin vs. jarjar ... it does the same thing but has better maven
integration - http://maven.apache.org/plugins/maven-shade-plugin/
On Thu, Oct 10, 2013 at 5:31 PM, Zhang Xiaoyu <[EMAIL PROTECTED]>wrote:

> Hi, Timothy,
> Thanks for your reply. Seems jarjar is a solution for me. I have a basic
> question to follow:
>
> I don't quite understand what is tag <keep> for ?
>
> My understand is
> 1.
> use <include> to grab the maven dependencies I want to re-package,
> 2.
> then use <pattern> and <result> in <rule> to define what class to
> re-package and what's the new name for re-packaged binary.
> 3.
> And use <keep> to indicate which class in the project will only use this
> re-packaged dependencies, in my case, it is JDBC class. Am I right ?
>
> below pom doesn't work for me, and JDBC class and other class still share
> the same thrift 0.9. Thrift 0.7 is omitted.
>
> Thanks a lot for help,
> Johnny
>
> <dependency>
>
> <groupId>org.apache.thrift</groupId>
>
> <artifactId>libthrift</artifactId>
>
> <version>0.9.0</version>
>
> </dependency>
>
> ......
>
> ......
>
> <plugin>
>
> <groupId>org.sonatype.plugins</groupId>
>
>       <artifactId>jarjar-maven-plugin</artifactId>
>
>       <executions>
>
>            <execution>
>
>              <phase>package</phase>
>
>              <goals>
>
>                <goal>jarjar</goal>
>
>              </goals>
>
>              <configuration>
>
>                <includes>
>
>                  <include>*org.apache.thrift:libthrift*</include>
>
>                </includes>
>
>                <rules>
>
>                  <rule>
>
>                    <pattern>org.apache.thrift.**</pattern>
>
>                    <result>com.xxx.JDBC.class.pattern.internal.thrift.@1</
> result>
>
>                  </rule>
>
>                  <keep>
>
>                    <pattern>com.xxx.JDBC.class.pattern.**</pattern>
>
>                   </keep>
>
>                </rules>
>
>              </configuration>
>
>            </execution>
>
>         </executions>
>
> </plugin>
>
>
> On Thu, Oct 10, 2013 at 12:28 PM, Timothy Potter <[EMAIL PROTECTED]>wrote:
>
>> Hi Zhang,
>>
>> I have the same issue in that I use some Cassandra client API's that
>> depend on Thrift 0.7 and HCatalog 0.11 that depends on Thrift 0.9. I opted
>> for using the jarjar utility to "shade" the thrift 0.9 classes. Here's what
>> I added to the build.xml file for the hcatalog-pig-adapter project:
>>
>> <target name="jarjar">
>>     <echo message="basedir=${basedir}"/>
>>     <copy file="${basedir}/../core/build/hcatalog-core-0.11.0.jar"
>> todir="${basedir}/deps"/>
>>     <copy file="${basedir}/build/hcatalog-pig-adapter-0.11.0.jar"
>> todir="${basedir}/deps"/>
>>     <copy todir="${basedir}/deps">
>>       <fileset dir="${basedir}/../../build/dist/lib"
>> includes="hive-*.jar"/>
>>     </copy>
>>     <taskdef name="jarjar" classname="com.tonicsystems.jarjar.JarJarTask"
>> classpath="lib/jarjar-1.4.jar"/>
>>     <jarjar jarfile="hcat-pig-shaded.jar">
>>         <zipgroupfileset dir="deps" includes="*.jar"/>
>>         <rule pattern="org.apache.thrift.**" result="org.shaded.thrift.@1
>> "/>
>>     </jarjar>
>> </target>
>>
>> I basically cooked up a deps directory that contains all the JARs that
>> the hcatalog-pig-adapter depends on and then used jarjar to create a single
>> jar with the thrift classes renamed at the bytecode level to
>> org.shaded.thrift
>>
>> You can do something similar for the JDBC code.
>>
>> Cheers,
>> Tim
>>
>>
>>
>> On Thu, Oct 10, 2013 at 1:19 PM, Edward Capriolo <[EMAIL PROTECTED]>wrote:
>>
>>> You are kinda screwed. Thrift is wire-compatible in many cases but not
>>> API compatible. You can not have two applications build off two versions of
>>> thrift in the same classpath without something like OSGI. To insulate the
>>> class loaders from each other.
>>>
>>> Your best bet is upgrading "other component" to thrift 9
>