Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hive, mail # user - Fwd: is that possible to use Hive 0.11 with libthrift 0.7


Copy link to this message
-
Fwd: is that possible to use Hive 0.11 with libthrift 0.7
Timothy Potter 2013-10-15, 01:34
Hi Johnny,

Sorry for the delayed response (was offline for a few days). So the code
that actually uses Thrift in is the Hive core JAR, which is what will get
shaded. The easiest thing to do is add this target to the root build.xml
file:

<target name="jarjar">
  <taskdef name="jarjar" classname="com.tonicsystems.jarjar.JarJarTask"
classpath="./jarjar-1.4.jar"/>
  <jarjar jarfile="hive-jdbc-shaded.jar">
      <zipgroupfileset dir="${hive.root}/build/dist/lib" includes="*.jar"/>
      <rule pattern="org.apache.thrift.**" result="org.shaded.thrift.@1"/>
  </jarjar>
</target>

Notice this assumes the JAR file for the jarjar task is in the hive.root
too. Run ant jarjar after doing a full ant clean package, which creates the
${hive.root}/build/dist/lib/*.jar

The jarjar task will produce a shaded JAR named hive-jdbc-shaded.jar that
contains all the hive jars and their dependencies with Thrift shaded into
the org.shaded.thrift package. Your client application needs to include
hive-jdbc-shaded.jar in its classpath (probably at the front).

Cheers,

Tim
On Thu, Oct 10, 2013 at 8:19 PM, Zhang Xiaoyu <[EMAIL PROTECTED]>wrote:

> Hi, Timothy,
> Thanks a lot for your reply. Appreciate it!
> I am facing the exact situation as yours. in one project, Cassandra client
> api need thrift 0.7 and Hive JDBC needs thrift 0.9......
>
> I am looking at the shade maven plugin, sorry I still have some silly
> questions, thanks a lot for your patience...
>
> I think you suggest two steps
> (1) need to include all of the dependencies of the Hive JDBC driver
> classes in a single JAR
> (2) the apply the shading the unified JAR file
>
> for *(1)* *instead* of putting the dependencies under <dependencies> tag,
> I now do this in the <plugins> section
>
> <plugin>
>
>         <groupId>org.apache.maven.plugins</groupId>
>
>         <artifactId>maven-shade-plugin</artifactId>
>
>         <executions>
>
>           <execution>
>
>             <phase>package</phase>
>
>             <goals>
>
>               <goal>shade</goal>
>
>             </goals>
>
>             <configuration>
>
>               <artifactSet>
>
>                 <includes>
>
>                   <include>org.apache.hive:hive-exec</include>
>
>                   <include>org.apache.hive:hive-jdbc</include>
>
>                   <include>org.apache.hive:hive-metastore</include>
>
>                   <include>org.apache.hive:hive-service</include>
>
>                   <include>org.apache.thrift:libfb303</include>
>
>                   <include>commons-logging:commons-logging</include>
>
>                   <include>org.apache.hadoop:hadoop-core</include>
>
>                 </includes>
>
>               </artifactSet>
>
>             </configuration>
>
>           </execution>
>
>         </executions>
>
>       </plugin>
>
>
> question here is how do I specify the version for each jars? for example I
> want hive-exec 0.9, where to put version number ?
>
> what will this shaded jar looks like , I search the ~/.m2/ folder, didn't
> find a jar with proper name.
>
> *for (2) *not sure how to do it, I mean how to specify only the Hive JDBC
> class refer to this shaded jar ? but Cassandra client API class doesn't use
> it?
>
> Sorry again for so many silly questions, thanks a lot for your time!
>
> Johnny
>
>
> On Thu, Oct 10, 2013 at 5:12 PM, Timothy Potter <[EMAIL PROTECTED]>wrote:
>
>> Sorry, sent too fast ... either way you do it, you'll need to include all
>> of the dependencies of the Hive JDBC driver classes in a single JAR and the
>> apply the shading the unified JAR file. This ensures that all classes that
>> depend on Thrift 0.9 get updated to use the shaded package. For example,
>> I'm working with the HCatalog Pig Adapter so I built a single JAR made up
>> of:
>>
>> antlr-runtime-3.4.jar       commons-io-2.4.jar hive-beeline-0.11.0.jar
>>      hive-serde-0.11.0.jar     jdo2-api-2.3-ec.jar
>> protobuf-java-2.4.1.jar
>> avro-1.7.1.jar       commons-lang-2.4.jar hive-cli-0.11.0.jar