Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hive, mail # user - is that possible to use Hive 0.11 with libthrift 0.7


Copy link to this message
-
Re: is that possible to use Hive 0.11 with libthrift 0.7
Timothy Potter 2013-10-11, 00:12
Sorry, sent too fast ... either way you do it, you'll need to include all
of the dependencies of the Hive JDBC driver classes in a single JAR and the
apply the shading the unified JAR file. This ensures that all classes that
depend on Thrift 0.9 get updated to use the shaded package. For example,
I'm working with the HCatalog Pig Adapter so I built a single JAR made up
of:

antlr-runtime-3.4.jar       commons-io-2.4.jar hive-beeline-0.11.0.jar
   hive-serde-0.11.0.jar     jdo2-api-2.3-ec.jar     protobuf-java-2.4.1.jar
avro-1.7.1.jar       commons-lang-2.4.jar hive-cli-0.11.0.jar
hive-service-0.11.0.jar     jetty-6.1.26.jar
servlet-api-2.5-20081211.jar
avro-mapred-1.7.1.jar       commons-logging-1.0.4.jar hive-common-0.11.0.jar
    hive-shims-0.11.0.jar     jetty-util-6.1.26.jar   slf4j-api-1.6.1.jar
commons-cli-1.2.jar       commons-logging-api-1.0.4.jar hive-contrib-0.11.0.jar
       jackson-core-asl-1.8.8.jar    jline-0.9.94.jar
slf4j-log4j12-1.6.1.jar
commons-codec-1.4.jar       commons-pool-1.5.4.jar hive-exec-0.11.0.jar
  jackson-jaxrs-1.8.8.jar     json-20090211.jar     snappy-0.2.jar
commons-collections-3.2.1.jar  derby-10.4.2.0.jar hive-hbase-handler-0.11.0.jar
 jackson-mapper-asl-1.8.8.jar  libfb303-0.9.0.jar      ST4-4.0.4.jar
commons-compress-1.4.1.jar     guava-11.0.2.jar hive-hwi-0.11.0.jar
jackson-xc-1.8.8.jar     libthrift-0.9.0.jar     tempus-fugit-1.1.jar
commons-configuration-1.6.jar  hcatalog-core-0.11.0.jar hive-jdbc-0.11.0.jar
    JavaEWAH-0.3.2.jar     log4j-1.2.16.jar     xz-1.0.jar
commons-dbcp-1.4.jar       hcatalog-pig-adapter-0.11.0.jar
hive-metastore-0.11.0.jar
     javolution-5.5.1.jar     metrics-core-2.1.2.jar  zookeeper-3.4.3.jar

Any class in any of these JAR files that depend on Thrift 0.9 classes will
get updated at the bytecode level to refer to my shaded package -
org.shaded.thrift

Make sense?
On Thu, Oct 10, 2013 at 6:04 PM, Timothy Potter <[EMAIL PROTECTED]>wrote:

> I'm not sure about keep, but if you're going to use maven, go with the
> shade plugin vs. jarjar ... it does the same thing but has better maven
> integration - http://maven.apache.org/plugins/maven-shade-plugin/
>
>
>
>
> On Thu, Oct 10, 2013 at 5:31 PM, Zhang Xiaoyu <[EMAIL PROTECTED]>wrote:
>
>> Hi, Timothy,
>> Thanks for your reply. Seems jarjar is a solution for me. I have a basic
>> question to follow:
>>
>> I don't quite understand what is tag <keep> for ?
>>
>> My understand is
>> 1.
>> use <include> to grab the maven dependencies I want to re-package,
>> 2.
>> then use <pattern> and <result> in <rule> to define what class to
>> re-package and what's the new name for re-packaged binary.
>> 3.
>> And use <keep> to indicate which class in the project will only use this
>> re-packaged dependencies, in my case, it is JDBC class. Am I right ?
>>
>> below pom doesn't work for me, and JDBC class and other class still share
>> the same thrift 0.9. Thrift 0.7 is omitted.
>>
>> Thanks a lot for help,
>> Johnny
>>
>> <dependency>
>>
>> <groupId>org.apache.thrift</groupId>
>>
>> <artifactId>libthrift</artifactId>
>>
>> <version>0.9.0</version>
>>
>> </dependency>
>>
>> ......
>>
>> ......
>>
>> <plugin>
>>
>> <groupId>org.sonatype.plugins</groupId>
>>
>>       <artifactId>jarjar-maven-plugin</artifactId>
>>
>>       <executions>
>>
>>            <execution>
>>
>>              <phase>package</phase>
>>
>>              <goals>
>>
>>                <goal>jarjar</goal>
>>
>>              </goals>
>>
>>              <configuration>
>>
>>                <includes>
>>
>>                  <include>*org.apache.thrift:libthrift*</include>
>>
>>                </includes>
>>
>>                <rules>
>>
>>                  <rule>
>>
>>                    <pattern>org.apache.thrift.**</pattern>
>>
>>                    <result>com.xxx.JDBC.class.pattern.internal.thrift.@1
>> </result>
>>
>>                  </rule>
>>
>>                  <keep>
>>
>>                    <pattern>com.xxx.JDBC.class.pattern.**</pattern>
>>
>>                   </keep>