Pradeep Kamath
2011-11-07, 18:21
Frank Maritato
2011-11-07, 18:47
Pradeep Kamath
2011-11-07, 19:25
Frank Maritato
2011-11-07, 19:38
Mark Kerzner
2011-11-07, 19:45
Mark Kerzner
2011-11-07, 19:48
Pradeep Kamath
2011-11-07, 19:55
Pradeep Kamath
2011-11-07, 19:54
-
Building dependencies into UDAF jarPradeep Kamath 2011-11-07, 18:21
Hi,
I am trying to build a hive UDAF which has dependencies on other
external jars (like log4j) - is there a way in which I can bundle the
main classes for my UDAF and all the dependent jars into one jar for use in my hive query. I tried having my UDF classes at the top level and
dependency jars in a "lib" directory inside one jar and then using "add
jar UDAF.jar" but that did not work (classes from dependency jars were
not being resolved). Setting HIVE_AUX_JARS_PATH environment variable
does work but requires that I have all the dependent jars separately on
the local filesystem where I run the hive query from. It would be more
convenient for me to build one self contained jar and refer to that in
my query (due to some restrictions in the cluster I run the query in) - is that possible?
Thanks,
Pradeep
I am trying to build a hive UDAF which has dependencies on other
external jars (like log4j) - is there a way in which I can bundle the
main classes for my UDAF and all the dependent jars into one jar for use in my hive query. I tried having my UDF classes at the top level and
dependency jars in a "lib" directory inside one jar and then using "add
jar UDAF.jar" but that did not work (classes from dependency jars were
not being resolved). Setting HIVE_AUX_JARS_PATH environment variable
does work but requires that I have all the dependent jars separately on
the local filesystem where I run the hive query from. It would be more
convenient for me to build one self contained jar and refer to that in
my query (due to some restrictions in the cluster I run the query in) - is that possible?
Thanks,
Pradeep
+
Pradeep Kamath 2011-11-07, 18:21
-
Re: Building dependencies into UDAF jarFrank Maritato 2011-11-07, 18:47
Hi Pradeep,
If you are using maven to build your code you can use the assembly plugin to produce a jar with dependencies.
http://maven.apache.org/plugins/maven-assembly-plugin/
Hope this helps
--
Frank Maritato
From: Pradeep Kamath <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>>
Reply-To: <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>>, Pradeep Kamath <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>>
Date: Mon, 7 Nov 2011 10:21:26 -0800
To: "[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>" <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>>
Subject: Building dependencies into UDAF jar
Hi,
I am trying to build a hive UDAF which has dependencies on other external jars (like log4j) - is there a way in which I can bundle the main classes for my UDAF and all the dependent jars into one jar for use in my hive query. I tried having my UDF classes at the top level and dependency jars in a "lib" directory inside one jar and then using "add jar UDAF.jar" but that did not work (classes from dependency jars were not being resolved). Setting HIVE_AUX_JARS_PATH environment variable does work but requires that I have all the dependent jars separately on the local filesystem where I run the hive query from. It would be more convenient for me to build one self contained jar and refer to that in my query (due to some restrictions in the cluster I run the query in) - is that possible?
Thanks,
Pradeep
If you are using maven to build your code you can use the assembly plugin to produce a jar with dependencies.
http://maven.apache.org/plugins/maven-assembly-plugin/
Hope this helps
--
Frank Maritato
From: Pradeep Kamath <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>>
Reply-To: <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>>, Pradeep Kamath <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>>
Date: Mon, 7 Nov 2011 10:21:26 -0800
To: "[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>" <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>>
Subject: Building dependencies into UDAF jar
Hi,
I am trying to build a hive UDAF which has dependencies on other external jars (like log4j) - is there a way in which I can bundle the main classes for my UDAF and all the dependent jars into one jar for use in my hive query. I tried having my UDF classes at the top level and dependency jars in a "lib" directory inside one jar and then using "add jar UDAF.jar" but that did not work (classes from dependency jars were not being resolved). Setting HIVE_AUX_JARS_PATH environment variable does work but requires that I have all the dependent jars separately on the local filesystem where I run the hive query from. It would be more convenient for me to build one self contained jar and refer to that in my query (due to some restrictions in the cluster I run the query in) - is that possible?
Thanks,
Pradeep
+
Frank Maritato 2011-11-07, 18:47
-
Re: Building dependencies into UDAF jarPradeep Kamath 2011-11-07, 19:25
Thanks for the suggestion Frank. Unfortunately I am using ant and am not very familiar with maven. Glancing through the link on maven-assembly-plugin, I am also not sure if the "assembly" created will work fine with hive/hadoop when the task runs on a remote node. Any way of creating a jar which hive/hadoop will understand to be a "composite" jar containing main classes and other dependent jars?
>________________________________
>From: Frank Maritato <[EMAIL PROTECTED]>
>To: "[EMAIL PROTECTED]" <[EMAIL PROTECTED]>; Pradeep Kamath <[EMAIL PROTECTED]>
>Sent: Monday, November 7, 2011 10:47 AM
>Subject: Re: Building dependencies into UDAF jar
>
>
>Hi Pradeep,
>
>
>If you are using maven to build your code you can use the assembly plugin to produce a jar with dependencies.
>http://maven.apache.org/plugins/maven-assembly-plugin/
>
>
>Hope this helps
>--
>Frank Maritato
>
>From: Pradeep Kamath <[EMAIL PROTECTED]>
>Reply-To: <[EMAIL PROTECTED]>, Pradeep Kamath <[EMAIL PROTECTED]>
>Date: Mon, 7 Nov 2011 10:21:26 -0800
>To: "[EMAIL PROTECTED]" <[EMAIL PROTECTED]>
>Subject: Building dependencies into UDAF jar
>
>
>
>Hi,
> I am trying to build a hive UDAF which has dependencies on other external jars (like log4j) - is there a way in which I can bundle the main classes for my UDAF and all the dependent jars into one jar for use in my hive query. I tried having my UDF classes at the top level and dependency jars in a "lib" directory inside one jar and then using "add jar UDAF.jar" but that did not work (classes from dependency jars were not being resolved). Setting HIVE_AUX_JARS_PATH environment variable does work but requires that I have all the dependent jars separately on the local filesystem where I run the hive query from. It would be more convenient for me to build one self contained jar and refer to that in my query (due to some restrictions in the cluster I run the query in) - is that possible?
>
>
>Thanks,
>Pradeep
>
>
>________________________________
>From: Frank Maritato <[EMAIL PROTECTED]>
>To: "[EMAIL PROTECTED]" <[EMAIL PROTECTED]>; Pradeep Kamath <[EMAIL PROTECTED]>
>Sent: Monday, November 7, 2011 10:47 AM
>Subject: Re: Building dependencies into UDAF jar
>
>
>Hi Pradeep,
>
>
>If you are using maven to build your code you can use the assembly plugin to produce a jar with dependencies.
>http://maven.apache.org/plugins/maven-assembly-plugin/
>
>
>Hope this helps
>--
>Frank Maritato
>
>From: Pradeep Kamath <[EMAIL PROTECTED]>
>Reply-To: <[EMAIL PROTECTED]>, Pradeep Kamath <[EMAIL PROTECTED]>
>Date: Mon, 7 Nov 2011 10:21:26 -0800
>To: "[EMAIL PROTECTED]" <[EMAIL PROTECTED]>
>Subject: Building dependencies into UDAF jar
>
>
>
>Hi,
> I am trying to build a hive UDAF which has dependencies on other external jars (like log4j) - is there a way in which I can bundle the main classes for my UDAF and all the dependent jars into one jar for use in my hive query. I tried having my UDF classes at the top level and dependency jars in a "lib" directory inside one jar and then using "add jar UDAF.jar" but that did not work (classes from dependency jars were not being resolved). Setting HIVE_AUX_JARS_PATH environment variable does work but requires that I have all the dependent jars separately on the local filesystem where I run the hive query from. It would be more convenient for me to build one self contained jar and refer to that in my query (due to some restrictions in the cluster I run the query in) - is that possible?
>
>
>Thanks,
>Pradeep
>
>
+
Pradeep Kamath 2011-11-07, 19:25
-
Re: Building dependencies into UDAF jarFrank Maritato 2011-11-07, 19:38
I'm not sure how to do this in ant, but it definitely works with maven+assembly because this is what we do for our production map reduce jobs and for our hive udf's. If you want I can probably dig up an example pom.xml that shows how to do this.
--
Frank Maritato
From: Pradeep Kamath <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>>
Reply-To: <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>>, Pradeep Kamath <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>>
Date: Mon, 7 Nov 2011 11:25:15 -0800
To: "[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>" <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>>
Subject: Re: Building dependencies into UDAF jar
Thanks for the suggestion Frank. Unfortunately I am using ant and am not very familiar with maven. Glancing through the link on maven-assembly-plugin, I am also not sure if the "assembly" created will work fine with hive/hadoop when the task runs on a remote node. Any way of creating a jar which hive/hadoop will understand to be a "composite" jar containing main classes and other dependent jars?
________________________________
From: Frank Maritato <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>>
To: "[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>" <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>>; Pradeep Kamath <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>>
Sent: Monday, November 7, 2011 10:47 AM
Subject: Re: Building dependencies into UDAF jar
Hi Pradeep,
If you are using maven to build your code you can use the assembly plugin to produce a jar with dependencies.
http://maven.apache.org/plugins/maven-assembly-plugin/
Hope this helps
--
Frank Maritato
From: Pradeep Kamath <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>>
Reply-To: <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>>, Pradeep Kamath <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>>
Date: Mon, 7 Nov 2011 10:21:26 -0800
To: "[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>" <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>>
Subject: Building dependencies into UDAF jar
Hi,
I am trying to build a hive UDAF which has dependencies on other external jars (like log4j) - is there a way in which I can bundle the main classes for my UDAF and all the dependent jars into one jar for use in my hive query. I tried having my UDF classes at the top level and dependency jars in a "lib" directory inside one jar and then using "add jar UDAF.jar" but that did not work (classes from dependency jars were not being resolved). Setting HIVE_AUX_JARS_PATH environment variable does work but requires that I have all the dependent jars separately on the local filesystem where I run the hive query from. It would be more convenient for me to build one self contained jar and refer to that in my query (due to some restrictions in the cluster I run the query in) - is that possible?
Thanks,
Pradeep
--
Frank Maritato
From: Pradeep Kamath <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>>
Reply-To: <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>>, Pradeep Kamath <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>>
Date: Mon, 7 Nov 2011 11:25:15 -0800
To: "[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>" <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>>
Subject: Re: Building dependencies into UDAF jar
Thanks for the suggestion Frank. Unfortunately I am using ant and am not very familiar with maven. Glancing through the link on maven-assembly-plugin, I am also not sure if the "assembly" created will work fine with hive/hadoop when the task runs on a remote node. Any way of creating a jar which hive/hadoop will understand to be a "composite" jar containing main classes and other dependent jars?
________________________________
From: Frank Maritato <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>>
To: "[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>" <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>>; Pradeep Kamath <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>>
Sent: Monday, November 7, 2011 10:47 AM
Subject: Re: Building dependencies into UDAF jar
Hi Pradeep,
If you are using maven to build your code you can use the assembly plugin to produce a jar with dependencies.
http://maven.apache.org/plugins/maven-assembly-plugin/
Hope this helps
--
Frank Maritato
From: Pradeep Kamath <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>>
Reply-To: <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>>, Pradeep Kamath <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>>
Date: Mon, 7 Nov 2011 10:21:26 -0800
To: "[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>" <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>>
Subject: Building dependencies into UDAF jar
Hi,
I am trying to build a hive UDAF which has dependencies on other external jars (like log4j) - is there a way in which I can bundle the main classes for my UDAF and all the dependent jars into one jar for use in my hive query. I tried having my UDF classes at the top level and dependency jars in a "lib" directory inside one jar and then using "add jar UDAF.jar" but that did not work (classes from dependency jars were not being resolved). Setting HIVE_AUX_JARS_PATH environment variable does work but requires that I have all the dependent jars separately on the local filesystem where I run the hive query from. It would be more convenient for me to build one self contained jar and refer to that in my query (due to some restrictions in the cluster I run the query in) - is that possible?
Thanks,
Pradeep
+
Frank Maritato 2011-11-07, 19:38
-
Re: Building dependencies into UDAF jarMark Kerzner 2011-11-07, 19:45
Here is an example of assembly plugin,
https://github.com/markkerzner/HadoopInPracticeCode/blob/master/pom.xml
Mark
On Mon, Nov 7, 2011 at 1:38 PM, Frank Maritato <[EMAIL PROTECTED]
> wrote:
> I'm not sure how to do this in ant, but it definitely works with
> maven+assembly because this is what we do for our production map reduce
> jobs and for our hive udf's. If you want I can probably dig up an example
> pom.xml that shows how to do this.
> --
> Frank Maritato
>
> From: Pradeep Kamath <[EMAIL PROTECTED]>
> Reply-To: <[EMAIL PROTECTED]>, Pradeep Kamath <[EMAIL PROTECTED]>
> Date: Mon, 7 Nov 2011 11:25:15 -0800
> To: "[EMAIL PROTECTED]" <[EMAIL PROTECTED]>
> Subject: Re: Building dependencies into UDAF jar
>
> Thanks for the suggestion Frank. Unfortunately I am using ant and am
> not very familiar with maven. Glancing through the link on
> maven-assembly-plugin, I am also not sure if the "assembly" created will
> work fine with hive/hadoop when the task runs on a remote node. Any way of
> creating a jar which hive/hadoop will understand to be a "composite" jar
> containing main classes and other dependent jars?
>
> ------------------------------
> *From:* Frank Maritato <[EMAIL PROTECTED]>
> *To:* "[EMAIL PROTECTED]" <[EMAIL PROTECTED]>; Pradeep Kamath <
> [EMAIL PROTECTED]>
> *Sent:* Monday, November 7, 2011 10:47 AM
> *Subject:* Re: Building dependencies into UDAF jar
>
> Hi Pradeep,
>
> If you are using maven to build your code you can use the assembly
> plugin to produce a jar with dependencies.
> http://maven.apache.org/plugins/maven-assembly-plugin/
>
> Hope this helps
> --
> Frank Maritato
>
> From: Pradeep Kamath <[EMAIL PROTECTED]>
> Reply-To: <[EMAIL PROTECTED]>, Pradeep Kamath <[EMAIL PROTECTED]>
> Date: Mon, 7 Nov 2011 10:21:26 -0800
> To: "[EMAIL PROTECTED]" <[EMAIL PROTECTED]>
> Subject: Building dependencies into UDAF jar
>
> Hi,
> I am trying to build a hive UDAF which has dependencies on other
> external jars (like log4j) - is there a way in which I can bundle the main
> classes for my UDAF and all the dependent jars into one jar for use in my
> hive query. I tried having my UDF classes at the top level and dependency
> jars in a "lib" directory inside one jar and then using "add jar UDAF.jar"
> but that did not work (classes from dependency jars were not being
> resolved). Setting HIVE_AUX_JARS_PATH environment variable does work but
> requires that I have all the dependent jars separately on the local
> filesystem where I run the hive query from. It would be more convenient for
> me to build one self contained jar and refer to that in my query (due to
> some restrictions in the cluster I run the query in) - is that possible?
>
> Thanks,
> Pradeep
>
>
>
https://github.com/markkerzner/HadoopInPracticeCode/blob/master/pom.xml
Mark
On Mon, Nov 7, 2011 at 1:38 PM, Frank Maritato <[EMAIL PROTECTED]
> wrote:
> I'm not sure how to do this in ant, but it definitely works with
> maven+assembly because this is what we do for our production map reduce
> jobs and for our hive udf's. If you want I can probably dig up an example
> pom.xml that shows how to do this.
> --
> Frank Maritato
>
> From: Pradeep Kamath <[EMAIL PROTECTED]>
> Reply-To: <[EMAIL PROTECTED]>, Pradeep Kamath <[EMAIL PROTECTED]>
> Date: Mon, 7 Nov 2011 11:25:15 -0800
> To: "[EMAIL PROTECTED]" <[EMAIL PROTECTED]>
> Subject: Re: Building dependencies into UDAF jar
>
> Thanks for the suggestion Frank. Unfortunately I am using ant and am
> not very familiar with maven. Glancing through the link on
> maven-assembly-plugin, I am also not sure if the "assembly" created will
> work fine with hive/hadoop when the task runs on a remote node. Any way of
> creating a jar which hive/hadoop will understand to be a "composite" jar
> containing main classes and other dependent jars?
>
> ------------------------------
> *From:* Frank Maritato <[EMAIL PROTECTED]>
> *To:* "[EMAIL PROTECTED]" <[EMAIL PROTECTED]>; Pradeep Kamath <
> [EMAIL PROTECTED]>
> *Sent:* Monday, November 7, 2011 10:47 AM
> *Subject:* Re: Building dependencies into UDAF jar
>
> Hi Pradeep,
>
> If you are using maven to build your code you can use the assembly
> plugin to produce a jar with dependencies.
> http://maven.apache.org/plugins/maven-assembly-plugin/
>
> Hope this helps
> --
> Frank Maritato
>
> From: Pradeep Kamath <[EMAIL PROTECTED]>
> Reply-To: <[EMAIL PROTECTED]>, Pradeep Kamath <[EMAIL PROTECTED]>
> Date: Mon, 7 Nov 2011 10:21:26 -0800
> To: "[EMAIL PROTECTED]" <[EMAIL PROTECTED]>
> Subject: Building dependencies into UDAF jar
>
> Hi,
> I am trying to build a hive UDAF which has dependencies on other
> external jars (like log4j) - is there a way in which I can bundle the main
> classes for my UDAF and all the dependent jars into one jar for use in my
> hive query. I tried having my UDF classes at the top level and dependency
> jars in a "lib" directory inside one jar and then using "add jar UDAF.jar"
> but that did not work (classes from dependency jars were not being
> resolved). Setting HIVE_AUX_JARS_PATH environment variable does work but
> requires that I have all the dependent jars separately on the local
> filesystem where I run the hive query from. It would be more convenient for
> me to build one self contained jar and refer to that in my query (due to
> some restrictions in the cluster I run the query in) - is that possible?
>
> Thanks,
> Pradeep
>
>
>
+
Mark Kerzner 2011-11-07, 19:45
-
Re: Building dependencies into UDAF jarMark Kerzner 2011-11-07, 19:48
To convert your project to maven, generate a maven project in Eclipse or
NetBeans (or IDEA :), and bring your source code into that projeuallct.
Actually,I have a hacked script for doing this in a shell, do you need it?
Mark
On Mon, Nov 7, 2011 at 1:45 PM, Mark Kerzner <[EMAIL PROTECTED]>wrote:
> Here is an example of assembly plugin,
> https://github.com/markkerzner/HadoopInPracticeCode/blob/master/pom.xml
>
> Mark
>
>
> On Mon, Nov 7, 2011 at 1:38 PM, Frank Maritato <
> [EMAIL PROTECTED]> wrote:
>
>> I'm not sure how to do this in ant, but it definitely works with
>> maven+assembly because this is what we do for our production map reduce
>> jobs and for our hive udf's. If you want I can probably dig up an example
>> pom.xml that shows how to do this.
>> --
>> Frank Maritato
>>
>> From: Pradeep Kamath <[EMAIL PROTECTED]>
>> Reply-To: <[EMAIL PROTECTED]>, Pradeep Kamath <[EMAIL PROTECTED]>
>> Date: Mon, 7 Nov 2011 11:25:15 -0800
>> To: "[EMAIL PROTECTED]" <[EMAIL PROTECTED]>
>> Subject: Re: Building dependencies into UDAF jar
>>
>> Thanks for the suggestion Frank. Unfortunately I am using ant and am
>> not very familiar with maven. Glancing through the link on
>> maven-assembly-plugin, I am also not sure if the "assembly" created will
>> work fine with hive/hadoop when the task runs on a remote node. Any way of
>> creating a jar which hive/hadoop will understand to be a "composite" jar
>> containing main classes and other dependent jars?
>>
>> ------------------------------
>> *From:* Frank Maritato <[EMAIL PROTECTED]>
>> *To:* "[EMAIL PROTECTED]" <[EMAIL PROTECTED]>; Pradeep Kamath <
>> [EMAIL PROTECTED]>
>> *Sent:* Monday, November 7, 2011 10:47 AM
>> *Subject:* Re: Building dependencies into UDAF jar
>>
>> Hi Pradeep,
>>
>> If you are using maven to build your code you can use the assembly
>> plugin to produce a jar with dependencies.
>> http://maven.apache.org/plugins/maven-assembly-plugin/
>>
>> Hope this helps
>> --
>> Frank Maritato
>>
>> From: Pradeep Kamath <[EMAIL PROTECTED]>
>> Reply-To: <[EMAIL PROTECTED]>, Pradeep Kamath <[EMAIL PROTECTED]>
>> Date: Mon, 7 Nov 2011 10:21:26 -0800
>> To: "[EMAIL PROTECTED]" <[EMAIL PROTECTED]>
>> Subject: Building dependencies into UDAF jar
>>
>> Hi,
>> I am trying to build a hive UDAF which has dependencies on other
>> external jars (like log4j) - is there a way in which I can bundle the main
>> classes for my UDAF and all the dependent jars into one jar for use in my
>> hive query. I tried having my UDF classes at the top level and dependency
>> jars in a "lib" directory inside one jar and then using "add jar UDAF.jar"
>> but that did not work (classes from dependency jars were not being
>> resolved). Setting HIVE_AUX_JARS_PATH environment variable does work but
>> requires that I have all the dependent jars separately on the local
>> filesystem where I run the hive query from. It would be more convenient for
>> me to build one self contained jar and refer to that in my query (due to
>> some restrictions in the cluster I run the query in) - is that possible?
>>
>> Thanks,
>> Pradeep
>>
>>
>>
>
NetBeans (or IDEA :), and bring your source code into that projeuallct.
Actually,I have a hacked script for doing this in a shell, do you need it?
Mark
On Mon, Nov 7, 2011 at 1:45 PM, Mark Kerzner <[EMAIL PROTECTED]>wrote:
> Here is an example of assembly plugin,
> https://github.com/markkerzner/HadoopInPracticeCode/blob/master/pom.xml
>
> Mark
>
>
> On Mon, Nov 7, 2011 at 1:38 PM, Frank Maritato <
> [EMAIL PROTECTED]> wrote:
>
>> I'm not sure how to do this in ant, but it definitely works with
>> maven+assembly because this is what we do for our production map reduce
>> jobs and for our hive udf's. If you want I can probably dig up an example
>> pom.xml that shows how to do this.
>> --
>> Frank Maritato
>>
>> From: Pradeep Kamath <[EMAIL PROTECTED]>
>> Reply-To: <[EMAIL PROTECTED]>, Pradeep Kamath <[EMAIL PROTECTED]>
>> Date: Mon, 7 Nov 2011 11:25:15 -0800
>> To: "[EMAIL PROTECTED]" <[EMAIL PROTECTED]>
>> Subject: Re: Building dependencies into UDAF jar
>>
>> Thanks for the suggestion Frank. Unfortunately I am using ant and am
>> not very familiar with maven. Glancing through the link on
>> maven-assembly-plugin, I am also not sure if the "assembly" created will
>> work fine with hive/hadoop when the task runs on a remote node. Any way of
>> creating a jar which hive/hadoop will understand to be a "composite" jar
>> containing main classes and other dependent jars?
>>
>> ------------------------------
>> *From:* Frank Maritato <[EMAIL PROTECTED]>
>> *To:* "[EMAIL PROTECTED]" <[EMAIL PROTECTED]>; Pradeep Kamath <
>> [EMAIL PROTECTED]>
>> *Sent:* Monday, November 7, 2011 10:47 AM
>> *Subject:* Re: Building dependencies into UDAF jar
>>
>> Hi Pradeep,
>>
>> If you are using maven to build your code you can use the assembly
>> plugin to produce a jar with dependencies.
>> http://maven.apache.org/plugins/maven-assembly-plugin/
>>
>> Hope this helps
>> --
>> Frank Maritato
>>
>> From: Pradeep Kamath <[EMAIL PROTECTED]>
>> Reply-To: <[EMAIL PROTECTED]>, Pradeep Kamath <[EMAIL PROTECTED]>
>> Date: Mon, 7 Nov 2011 10:21:26 -0800
>> To: "[EMAIL PROTECTED]" <[EMAIL PROTECTED]>
>> Subject: Building dependencies into UDAF jar
>>
>> Hi,
>> I am trying to build a hive UDAF which has dependencies on other
>> external jars (like log4j) - is there a way in which I can bundle the main
>> classes for my UDAF and all the dependent jars into one jar for use in my
>> hive query. I tried having my UDF classes at the top level and dependency
>> jars in a "lib" directory inside one jar and then using "add jar UDAF.jar"
>> but that did not work (classes from dependency jars were not being
>> resolved). Setting HIVE_AUX_JARS_PATH environment variable does work but
>> requires that I have all the dependent jars separately on the local
>> filesystem where I run the hive query from. It would be more convenient for
>> me to build one self contained jar and refer to that in my query (due to
>> some restrictions in the cluster I run the query in) - is that possible?
>>
>> Thanks,
>> Pradeep
>>
>>
>>
>
+
Mark Kerzner 2011-11-07, 19:48
-
Re: Building dependencies into UDAF jarPradeep Kamath 2011-11-07, 19:55
Yes - please do share the script - appreciate the help!
>________________________________
>From: Mark Kerzner <[EMAIL PROTECTED]>
>To: [EMAIL PROTECTED]
>Cc: Pradeep Kamath <[EMAIL PROTECTED]>
>Sent: Monday, November 7, 2011 11:48 AM
>Subject: Re: Building dependencies into UDAF jar
>
>
>To convert your project to maven, generate a maven project in Eclipse or NetBeans (or IDEA :), and bring your source code into that projeuallct.
>
>Actually,I have a hacked script for doing this in a shell, do you need it?
>
>Mark
>
>
>On Mon, Nov 7, 2011 at 1:45 PM, Mark Kerzner <[EMAIL PROTECTED]> wrote:
>
>Here is an example of assembly plugin, https://github.com/markkerzner/HadoopInPracticeCode/blob/master/pom.xml
>>
>>Mark
>>
>>
>>
>>On Mon, Nov 7, 2011 at 1:38 PM, Frank Maritato <[EMAIL PROTECTED]> wrote:
>>
>>I'm not sure how to do this in ant, but it definitely works with maven+assembly because this is what we do for our production map reduce jobs and for our hive udf's. If you want I can probably dig up an example pom.xml that shows how to do this.
>>>--
>>>Frank Maritato
>>>
>>>From: Pradeep Kamath <[EMAIL PROTECTED]>
>>>Reply-To: <[EMAIL PROTECTED]>, Pradeep Kamath <[EMAIL PROTECTED]>
>>>Date: Mon, 7 Nov 2011 11:25:15 -0800
>>>To: "[EMAIL PROTECTED]" <[EMAIL PROTECTED]>
>>>Subject: Re: Building dependencies into UDAF jar
>>>
>>>
>>>
>>>Thanks for the suggestion Frank. Unfortunately I am using ant and am not very familiar with maven. Glancing through the link on maven-assembly-plugin, I am also not sure if the "assembly" created will work fine with hive/hadoop when the task runs on a remote node. Any way of creating a jar which hive/hadoop will understand to be a "composite" jar containing main classes and other dependent jars?
>>>
>>>
>>>
>>>
>>>>________________________________
>>>> From: Frank Maritato <[EMAIL PROTECTED]>
>>>>To: "[EMAIL PROTECTED]" <[EMAIL PROTECTED]>; Pradeep Kamath <[EMAIL PROTECTED]>
>>>>Sent: Monday, November 7, 2011 10:47 AM
>>>>Subject: Re: Building dependencies into UDAF jar
>>>>
>>>>
>>>>Hi Pradeep,
>>>>
>>>>
>>>>If you are using maven to build your code you can use the assembly plugin to produce a jar with dependencies.
>>>>http://maven.apache.org/plugins/maven-assembly-plugin/
>>>>
>>>>
>>>>Hope this helps
>>>>--
>>>>Frank Maritato
>>>>
>>>>From: Pradeep Kamath <[EMAIL PROTECTED]>
>>>>Reply-To: <[EMAIL PROTECTED]>, Pradeep Kamath <[EMAIL PROTECTED]>
>>>>Date: Mon, 7 Nov 2011 10:21:26 -0800
>>>>To: "[EMAIL PROTECTED]" <[EMAIL PROTECTED]>
>>>>Subject: Building dependencies into UDAF jar
>>>>
>>>>
>>>>
>>>>Hi,
>>>> I am trying to build a hive UDAF which has dependencies on other external jars (like log4j) - is there a way in which I can bundle the main classes for my UDAF and all the dependent jars into one jar for use in my hive query. I tried having my UDF classes at the top level and dependency jars in a "lib" directory inside one jar and then using "add jar UDAF.jar" but that did not work (classes from dependency jars were not being resolved). Setting HIVE_AUX_JARS_PATH environment variable does work but requires that I have all the dependent jars separately on the local filesystem where I run the hive query from. It would be more convenient for me to build one self contained jar and refer to that in my query (due to some restrictions in the cluster I run the query in) - is that possible?
>>>>
>>>>
>>>>Thanks,
>>>>Pradeep
>>>>
>>>>
>>
>
>
>
>________________________________
>From: Mark Kerzner <[EMAIL PROTECTED]>
>To: [EMAIL PROTECTED]
>Cc: Pradeep Kamath <[EMAIL PROTECTED]>
>Sent: Monday, November 7, 2011 11:48 AM
>Subject: Re: Building dependencies into UDAF jar
>
>
>To convert your project to maven, generate a maven project in Eclipse or NetBeans (or IDEA :), and bring your source code into that projeuallct.
>
>Actually,I have a hacked script for doing this in a shell, do you need it?
>
>Mark
>
>
>On Mon, Nov 7, 2011 at 1:45 PM, Mark Kerzner <[EMAIL PROTECTED]> wrote:
>
>Here is an example of assembly plugin, https://github.com/markkerzner/HadoopInPracticeCode/blob/master/pom.xml
>>
>>Mark
>>
>>
>>
>>On Mon, Nov 7, 2011 at 1:38 PM, Frank Maritato <[EMAIL PROTECTED]> wrote:
>>
>>I'm not sure how to do this in ant, but it definitely works with maven+assembly because this is what we do for our production map reduce jobs and for our hive udf's. If you want I can probably dig up an example pom.xml that shows how to do this.
>>>--
>>>Frank Maritato
>>>
>>>From: Pradeep Kamath <[EMAIL PROTECTED]>
>>>Reply-To: <[EMAIL PROTECTED]>, Pradeep Kamath <[EMAIL PROTECTED]>
>>>Date: Mon, 7 Nov 2011 11:25:15 -0800
>>>To: "[EMAIL PROTECTED]" <[EMAIL PROTECTED]>
>>>Subject: Re: Building dependencies into UDAF jar
>>>
>>>
>>>
>>>Thanks for the suggestion Frank. Unfortunately I am using ant and am not very familiar with maven. Glancing through the link on maven-assembly-plugin, I am also not sure if the "assembly" created will work fine with hive/hadoop when the task runs on a remote node. Any way of creating a jar which hive/hadoop will understand to be a "composite" jar containing main classes and other dependent jars?
>>>
>>>
>>>
>>>
>>>>________________________________
>>>> From: Frank Maritato <[EMAIL PROTECTED]>
>>>>To: "[EMAIL PROTECTED]" <[EMAIL PROTECTED]>; Pradeep Kamath <[EMAIL PROTECTED]>
>>>>Sent: Monday, November 7, 2011 10:47 AM
>>>>Subject: Re: Building dependencies into UDAF jar
>>>>
>>>>
>>>>Hi Pradeep,
>>>>
>>>>
>>>>If you are using maven to build your code you can use the assembly plugin to produce a jar with dependencies.
>>>>http://maven.apache.org/plugins/maven-assembly-plugin/
>>>>
>>>>
>>>>Hope this helps
>>>>--
>>>>Frank Maritato
>>>>
>>>>From: Pradeep Kamath <[EMAIL PROTECTED]>
>>>>Reply-To: <[EMAIL PROTECTED]>, Pradeep Kamath <[EMAIL PROTECTED]>
>>>>Date: Mon, 7 Nov 2011 10:21:26 -0800
>>>>To: "[EMAIL PROTECTED]" <[EMAIL PROTECTED]>
>>>>Subject: Building dependencies into UDAF jar
>>>>
>>>>
>>>>
>>>>Hi,
>>>> I am trying to build a hive UDAF which has dependencies on other external jars (like log4j) - is there a way in which I can bundle the main classes for my UDAF and all the dependent jars into one jar for use in my hive query. I tried having my UDF classes at the top level and dependency jars in a "lib" directory inside one jar and then using "add jar UDAF.jar" but that did not work (classes from dependency jars were not being resolved). Setting HIVE_AUX_JARS_PATH environment variable does work but requires that I have all the dependent jars separately on the local filesystem where I run the hive query from. It would be more convenient for me to build one self contained jar and refer to that in my query (due to some restrictions in the cluster I run the query in) - is that possible?
>>>>
>>>>
>>>>Thanks,
>>>>Pradeep
>>>>
>>>>
>>
>
>
>
+
Pradeep Kamath 2011-11-07, 19:55
-
Re: Building dependencies into UDAF jarPradeep Kamath 2011-11-07, 19:54
Thanks Frank - yes, please do share the pom.xml - will try and figure out how to do it in ant - also if you can share how the assembly jar looks like (ex: where do the dependency jars sit, is there a custom classloader involved) etc it might help me replicate it in ant - if its too much trouble then just a pom.xml can be a good starting point for me to try.
If others have any ant friendly mechanisms for this, would greatly appreciate it.
Thanks,
Pradeep
>________________________________
>From: Frank Maritato <[EMAIL PROTECTED]>
>To: "[EMAIL PROTECTED]" <[EMAIL PROTECTED]>; Pradeep Kamath <[EMAIL PROTECTED]>
>Sent: Monday, November 7, 2011 11:38 AM
>Subject: Re: Building dependencies into UDAF jar
>
>
>I'm not sure how to do this in ant, but it definitely works with maven+assembly because this is what we do for our production map reduce jobs and for our hive udf's. If you want I can probably dig up an example pom.xml that shows how to do this.
>--
>Frank Maritato
>
>From: Pradeep Kamath <[EMAIL PROTECTED]>
>Reply-To: <[EMAIL PROTECTED]>, Pradeep Kamath <[EMAIL PROTECTED]>
>Date: Mon, 7 Nov 2011 11:25:15 -0800
>To: "[EMAIL PROTECTED]" <[EMAIL PROTECTED]>
>Subject: Re: Building dependencies into UDAF jar
>
>
>
>Thanks for the suggestion Frank. Unfortunately I am using ant and am not very familiar with maven. Glancing through the link on maven-assembly-plugin, I am also not sure if the "assembly" created will work fine with hive/hadoop when the task runs on a remote node. Any way of creating a jar which hive/hadoop will understand to be a "composite" jar containing main classes and other dependent jars?
>
>
>
>
>>________________________________
>> From: Frank Maritato <[EMAIL PROTECTED]>
>>To: "[EMAIL PROTECTED]" <[EMAIL PROTECTED]>; Pradeep Kamath <[EMAIL PROTECTED]>
>>Sent: Monday, November 7, 2011 10:47 AM
>>Subject: Re: Building dependencies into UDAF jar
>>
>>
>>Hi Pradeep,
>>
>>
>>If you are using maven to build your code you can use the assembly plugin to produce a jar with dependencies.
>>http://maven.apache.org/plugins/maven-assembly-plugin/
>>
>>
>>Hope this helps
>>--
>>Frank Maritato
>>
>>From: Pradeep Kamath <[EMAIL PROTECTED]>
>>Reply-To: <[EMAIL PROTECTED]>, Pradeep Kamath <[EMAIL PROTECTED]>
>>Date: Mon, 7 Nov 2011 10:21:26 -0800
>>To: "[EMAIL PROTECTED]" <[EMAIL PROTECTED]>
>>Subject: Building dependencies into UDAF jar
>>
>>
>>
>>Hi,
>> I am trying to build a hive UDAF which has dependencies on other external jars (like log4j) - is there a way in which I can bundle the main classes for my UDAF and all the dependent jars into one jar for use in my hive query. I tried having my UDF classes at the top level and dependency jars in a "lib" directory inside one jar and then using "add jar UDAF.jar" but that did not work (classes from dependency jars were not being resolved). Setting HIVE_AUX_JARS_PATH environment variable does work but requires that I have all the dependent jars separately on the local filesystem where I run the hive query from. It would be more convenient for me to build one self contained jar and refer to that in my query (due to some restrictions in the cluster I run the query in) - is that possible?
>>
>>
>>Thanks,
>>Pradeep
>>
>>
>
>
If others have any ant friendly mechanisms for this, would greatly appreciate it.
Thanks,
Pradeep
>________________________________
>From: Frank Maritato <[EMAIL PROTECTED]>
>To: "[EMAIL PROTECTED]" <[EMAIL PROTECTED]>; Pradeep Kamath <[EMAIL PROTECTED]>
>Sent: Monday, November 7, 2011 11:38 AM
>Subject: Re: Building dependencies into UDAF jar
>
>
>I'm not sure how to do this in ant, but it definitely works with maven+assembly because this is what we do for our production map reduce jobs and for our hive udf's. If you want I can probably dig up an example pom.xml that shows how to do this.
>--
>Frank Maritato
>
>From: Pradeep Kamath <[EMAIL PROTECTED]>
>Reply-To: <[EMAIL PROTECTED]>, Pradeep Kamath <[EMAIL PROTECTED]>
>Date: Mon, 7 Nov 2011 11:25:15 -0800
>To: "[EMAIL PROTECTED]" <[EMAIL PROTECTED]>
>Subject: Re: Building dependencies into UDAF jar
>
>
>
>Thanks for the suggestion Frank. Unfortunately I am using ant and am not very familiar with maven. Glancing through the link on maven-assembly-plugin, I am also not sure if the "assembly" created will work fine with hive/hadoop when the task runs on a remote node. Any way of creating a jar which hive/hadoop will understand to be a "composite" jar containing main classes and other dependent jars?
>
>
>
>
>>________________________________
>> From: Frank Maritato <[EMAIL PROTECTED]>
>>To: "[EMAIL PROTECTED]" <[EMAIL PROTECTED]>; Pradeep Kamath <[EMAIL PROTECTED]>
>>Sent: Monday, November 7, 2011 10:47 AM
>>Subject: Re: Building dependencies into UDAF jar
>>
>>
>>Hi Pradeep,
>>
>>
>>If you are using maven to build your code you can use the assembly plugin to produce a jar with dependencies.
>>http://maven.apache.org/plugins/maven-assembly-plugin/
>>
>>
>>Hope this helps
>>--
>>Frank Maritato
>>
>>From: Pradeep Kamath <[EMAIL PROTECTED]>
>>Reply-To: <[EMAIL PROTECTED]>, Pradeep Kamath <[EMAIL PROTECTED]>
>>Date: Mon, 7 Nov 2011 10:21:26 -0800
>>To: "[EMAIL PROTECTED]" <[EMAIL PROTECTED]>
>>Subject: Building dependencies into UDAF jar
>>
>>
>>
>>Hi,
>> I am trying to build a hive UDAF which has dependencies on other external jars (like log4j) - is there a way in which I can bundle the main classes for my UDAF and all the dependent jars into one jar for use in my hive query. I tried having my UDF classes at the top level and dependency jars in a "lib" directory inside one jar and then using "add jar UDAF.jar" but that did not work (classes from dependency jars were not being resolved). Setting HIVE_AUX_JARS_PATH environment variable does work but requires that I have all the dependent jars separately on the local filesystem where I run the hive query from. It would be more convenient for me to build one self contained jar and refer to that in my query (due to some restrictions in the cluster I run the query in) - is that possible?
>>
>>
>>Thanks,
>>Pradeep
>>
>>
>
>
+
Pradeep Kamath 2011-11-07, 19:54
