Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hive >> mail # user >> Building dependencies into UDAF jar


Copy link to this message
-
Re: Building dependencies into UDAF jar
Thanks Frank - yes, please do share the pom.xml - will try and figure out how to do it in ant - also if you can share how the assembly jar looks like (ex: where do the dependency jars sit, is there a custom classloader involved) etc it might help me replicate it in ant - if its too much trouble then just a pom.xml can be a good starting point for me to try.

If others have any ant friendly mechanisms for this, would greatly appreciate it.

Thanks,
Pradeep
>________________________________
>From: Frank Maritato <[EMAIL PROTECTED]>
>To: "[EMAIL PROTECTED]" <[EMAIL PROTECTED]>; Pradeep Kamath <[EMAIL PROTECTED]>
>Sent: Monday, November 7, 2011 11:38 AM
>Subject: Re: Building dependencies into UDAF jar
>
>
>I'm not sure how to do this in ant, but it definitely works with maven+assembly because this is what we do for our production map reduce jobs and for our hive udf's. If you want I can probably dig up an example pom.xml that shows how to do this.
>-- 
>Frank Maritato
>
>From: Pradeep Kamath <[EMAIL PROTECTED]>
>Reply-To: <[EMAIL PROTECTED]>, Pradeep Kamath <[EMAIL PROTECTED]>
>Date: Mon, 7 Nov 2011 11:25:15 -0800
>To: "[EMAIL PROTECTED]" <[EMAIL PROTECTED]>
>Subject: Re: Building dependencies into UDAF jar
>
>
>
>Thanks for the suggestion Frank. Unfortunately I am using ant and am not very familiar with maven. Glancing through the link on maven-assembly-plugin, I am also not sure if the "assembly" created will work fine with hive/hadoop when the task runs on a remote node. Any way of creating a jar which hive/hadoop will understand to be a "composite" jar containing main classes and other dependent jars?
>
>
>
>
>>________________________________
>> From: Frank Maritato <[EMAIL PROTECTED]>
>>To: "[EMAIL PROTECTED]" <[EMAIL PROTECTED]>; Pradeep Kamath <[EMAIL PROTECTED]>
>>Sent: Monday, November 7, 2011 10:47 AM
>>Subject: Re: Building dependencies into UDAF jar
>>
>>
>>Hi Pradeep,
>>
>>
>>If you are using maven to build your code you can use the assembly plugin to produce a jar with dependencies. 
>>http://maven.apache.org/plugins/maven-assembly-plugin/
>>
>>
>>Hope this helps
>>-- 
>>Frank Maritato
>>
>>From: Pradeep Kamath <[EMAIL PROTECTED]>
>>Reply-To: <[EMAIL PROTECTED]>, Pradeep Kamath <[EMAIL PROTECTED]>
>>Date: Mon, 7 Nov 2011 10:21:26 -0800
>>To: "[EMAIL PROTECTED]" <[EMAIL PROTECTED]>
>>Subject: Building dependencies into UDAF jar
>>
>>
>>
>>Hi,
>>  I am trying to build a hive UDAF which has dependencies on other external jars (like log4j) - is there a way in which I can bundle the main classes for my UDAF and all the dependent jars into one jar for use in my hive query. I tried having my UDF classes at the top level and dependency jars in a "lib" directory inside one jar and then using "add jar UDAF.jar" but that did not work (classes from dependency jars were not being resolved). Setting HIVE_AUX_JARS_PATH environment variable does work but requires that I have all the dependent jars separately on the local filesystem where I run the hive query from. It would be more convenient for me to build one self contained jar and refer to that in my query (due to some restrictions in the cluster I run the query in) - is that possible?
>>
>>
>>Thanks,
>>Pradeep
>>
>>
>
>
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB