Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hive, mail # user - Building dependencies into UDAF jar


Copy link to this message
-
Re: Building dependencies into UDAF jar
Pradeep Kamath 2011-11-07, 19:25
Thanks for the suggestion Frank. Unfortunately I am using ant and am not very familiar with maven. Glancing through the link on maven-assembly-plugin, I am also not sure if the "assembly" created will work fine with hive/hadoop when the task runs on a remote node. Any way of creating a jar which hive/hadoop will understand to be a "composite" jar containing main classes and other dependent jars?
>________________________________
>From: Frank Maritato <[EMAIL PROTECTED]>
>To: "[EMAIL PROTECTED]" <[EMAIL PROTECTED]>; Pradeep Kamath <[EMAIL PROTECTED]>
>Sent: Monday, November 7, 2011 10:47 AM
>Subject: Re: Building dependencies into UDAF jar
>
>
>Hi Pradeep,
>
>
>If you are using maven to build your code you can use the assembly plugin to produce a jar with dependencies. 
>http://maven.apache.org/plugins/maven-assembly-plugin/
>
>
>Hope this helps
>-- 
>Frank Maritato
>
>From: Pradeep Kamath <[EMAIL PROTECTED]>
>Reply-To: <[EMAIL PROTECTED]>, Pradeep Kamath <[EMAIL PROTECTED]>
>Date: Mon, 7 Nov 2011 10:21:26 -0800
>To: "[EMAIL PROTECTED]" <[EMAIL PROTECTED]>
>Subject: Building dependencies into UDAF jar
>
>
>
>Hi,
>  I am trying to build a hive UDAF which has dependencies on other external jars (like log4j) - is there a way in which I can bundle the main classes for my UDAF and all the dependent jars into one jar for use in my hive query. I tried having my UDF classes at the top level and dependency jars in a "lib" directory inside one jar and then using "add jar UDAF.jar" but that did not work (classes from dependency jars were not being resolved). Setting HIVE_AUX_JARS_PATH environment variable does work but requires that I have all the dependent jars separately on the local filesystem where I run the hive query from. It would be more convenient for me to build one self contained jar and refer to that in my query (due to some restrictions in the cluster I run the query in) - is that possible?
>
>
>Thanks,
>Pradeep
>
>