Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Hive >> mail # user >> Building dependencies into UDAF jar


+
Pradeep Kamath 2011-11-07, 18:21
+
Frank Maritato 2011-11-07, 18:47
+
Pradeep Kamath 2011-11-07, 19:25
+
Frank Maritato 2011-11-07, 19:38
+
Mark Kerzner 2011-11-07, 19:45
+
Mark Kerzner 2011-11-07, 19:48
Copy link to this message
-
Re: Building dependencies into UDAF jar
Yes - please do share the script - appreciate the help!
>________________________________
>From: Mark Kerzner <[EMAIL PROTECTED]>
>To: [EMAIL PROTECTED]
>Cc: Pradeep Kamath <[EMAIL PROTECTED]>
>Sent: Monday, November 7, 2011 11:48 AM
>Subject: Re: Building dependencies into UDAF jar
>
>
>To convert your project to maven, generate a maven project in Eclipse or NetBeans (or IDEA :), and bring your source code into that projeuallct.
>
>Actually,I have a hacked script for doing this in a shell, do you need it?
>
>Mark
>
>
>On Mon, Nov 7, 2011 at 1:45 PM, Mark Kerzner <[EMAIL PROTECTED]> wrote:
>
>Here is an example of assembly plugin, https://github.com/markkerzner/HadoopInPracticeCode/blob/master/pom.xml
>>
>>Mark
>>
>> 
>>
>>On Mon, Nov 7, 2011 at 1:38 PM, Frank Maritato <[EMAIL PROTECTED]> wrote:
>>
>>I'm not sure how to do this in ant, but it definitely works with maven+assembly because this is what we do for our production map reduce jobs and for our hive udf's. If you want I can probably dig up an example pom.xml that shows how to do this.
>>>-- 
>>>Frank Maritato
>>>
>>>From: Pradeep Kamath <[EMAIL PROTECTED]>
>>>Reply-To: <[EMAIL PROTECTED]>, Pradeep Kamath <[EMAIL PROTECTED]>
>>>Date: Mon, 7 Nov 2011 11:25:15 -0800
>>>To: "[EMAIL PROTECTED]" <[EMAIL PROTECTED]>
>>>Subject: Re: Building dependencies into UDAF jar
>>>
>>>
>>>
>>>Thanks for the suggestion Frank. Unfortunately I am using ant and am not very familiar with maven. Glancing through the link on maven-assembly-plugin, I am also not sure if the "assembly" created will work fine with hive/hadoop when the task runs on a remote node. Any way of creating a jar which hive/hadoop will understand to be a "composite" jar containing main classes and other dependent jars?
>>>
>>>
>>>
>>>
>>>>________________________________
>>>> From: Frank Maritato <[EMAIL PROTECTED]>
>>>>To: "[EMAIL PROTECTED]" <[EMAIL PROTECTED]>; Pradeep Kamath <[EMAIL PROTECTED]>
>>>>Sent: Monday, November 7, 2011 10:47 AM
>>>>Subject: Re: Building dependencies into UDAF jar
>>>>
>>>>
>>>>Hi Pradeep,
>>>>
>>>>
>>>>If you are using maven to build your code you can use the assembly plugin to produce a jar with dependencies. 
>>>>http://maven.apache.org/plugins/maven-assembly-plugin/
>>>>
>>>>
>>>>Hope this helps
>>>>-- 
>>>>Frank Maritato
>>>>
>>>>From: Pradeep Kamath <[EMAIL PROTECTED]>
>>>>Reply-To: <[EMAIL PROTECTED]>, Pradeep Kamath <[EMAIL PROTECTED]>
>>>>Date: Mon, 7 Nov 2011 10:21:26 -0800
>>>>To: "[EMAIL PROTECTED]" <[EMAIL PROTECTED]>
>>>>Subject: Building dependencies into UDAF jar
>>>>
>>>>
>>>>
>>>>Hi,
>>>>  I am trying to build a hive UDAF which has dependencies on other external jars (like log4j) - is there a way in which I can bundle the main classes for my UDAF and all the dependent jars into one jar for use in my hive query. I tried having my UDF classes at the top level and dependency jars in a "lib" directory inside one jar and then using "add jar UDAF.jar" but that did not work (classes from dependency jars were not being resolved). Setting HIVE_AUX_JARS_PATH environment variable does work but requires that I have all the dependent jars separately on the local filesystem where I run the hive query from. It would be more convenient for me to build one self contained jar and refer to that in my query (due to some restrictions in the cluster I run the query in) - is that possible?
>>>>
>>>>
>>>>Thanks,
>>>>Pradeep
>>>>
>>>>
>>
>
>
>
+
Pradeep Kamath 2011-11-07, 19:54
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB