Utkarsh Gupta
2012-04-04, 06:52
Devaraj k
2012-04-04, 07:04
Utkarsh Gupta
2012-04-04, 07:25
Bejoy Ks
2012-04-04, 07:36
Devaraj k
2012-04-04, 08:38
Utkarsh Gupta
2012-04-04, 09:59
Utkarsh Gupta
2012-04-04, 11:55
Harsh J
2012-04-04, 12:49
Utkarsh Gupta
2012-04-04, 12:55
Harsh J
2012-04-04, 13:01
Utkarsh Gupta
2012-04-04, 13:13
Ioan Eugen Stan
2012-04-04, 15:06
GUOJUN Zhu
2012-04-04, 17:13
-
Including third party jar files in Map Reduce jobUtkarsh Gupta 2012-04-04, 06:52
Hi All,
I am new to Hadoop and was trying to generate random numbers using apache commons math library.
I used Netbeans to build the jar file and the manifest has path to commons-math jar as lib/commons-math3.jar
I have placed this jar file in HADOOP_HOME/lib folder but still I am getting ClassNotFoundException.
I tried using -libjar option with $HADOOP_HOME/bin/Hadoop jar myprg.jar <inputpath> <outputpath> -libjar <jarpath>
And $HADOOP_HOME/bin/Hadoop jar myprg.jar -libjar <jarpath> <inputpath> <outputpath>
But this is not working. Please help.
Thanks and Regards
Utkarsh Gupta
**************** CAUTION - Disclaimer *****************
This e-mail contains PRIVILEGED AND CONFIDENTIAL INFORMATION intended solely
for the use of the addressee(s). If you are not the intended recipient, please
notify the sender by e-mail and delete the original message. Further, you are not
to copy, disclose, or distribute this e-mail or its contents to any other person and
any such actions are unlawful. This e-mail may contain viruses. Infosys has taken
every reasonable precaution to minimize this risk, but is not liable for any damage
you may sustain as a result of any virus in this e-mail. You should carry out your
own virus checks before opening the e-mail or attachment. Infosys reserves the
right to monitor and review the content of all messages sent to or from this e-mail
address. Messages sent to or from this e-mail address may be stored on the
Infosys e-mail system.
***INFOSYS******** End of Disclaimer ********INFOSYS***
I am new to Hadoop and was trying to generate random numbers using apache commons math library.
I used Netbeans to build the jar file and the manifest has path to commons-math jar as lib/commons-math3.jar
I have placed this jar file in HADOOP_HOME/lib folder but still I am getting ClassNotFoundException.
I tried using -libjar option with $HADOOP_HOME/bin/Hadoop jar myprg.jar <inputpath> <outputpath> -libjar <jarpath>
And $HADOOP_HOME/bin/Hadoop jar myprg.jar -libjar <jarpath> <inputpath> <outputpath>
But this is not working. Please help.
Thanks and Regards
Utkarsh Gupta
**************** CAUTION - Disclaimer *****************
This e-mail contains PRIVILEGED AND CONFIDENTIAL INFORMATION intended solely
for the use of the addressee(s). If you are not the intended recipient, please
notify the sender by e-mail and delete the original message. Further, you are not
to copy, disclose, or distribute this e-mail or its contents to any other person and
any such actions are unlawful. This e-mail may contain viruses. Infosys has taken
every reasonable precaution to minimize this risk, but is not liable for any damage
you may sustain as a result of any virus in this e-mail. You should carry out your
own virus checks before opening the e-mail or attachment. Infosys reserves the
right to monitor and review the content of all messages sent to or from this e-mail
address. Messages sent to or from this e-mail address may be stored on the
Infosys e-mail system.
***INFOSYS******** End of Disclaimer ********INFOSYS***
-
RE: Including third party jar files in Map Reduce jobDevaraj k 2012-04-04, 07:04
Hi Utkarsh,
The usage of the jar command is like this,
Usage: hadoop jar <jar> [mainClass] args...
If you want the commons-math3.jar to be available for all the tasks you can do any one of these 1. Copy the jar file in $HADOOP_HOME/lib dir or 2. Use the generic option -libjars.
Can you give the stack trace of your problem for which class it is giving ClassNotFoundException(i.e for main class or math lib class)?
Thanks
Devaraj
________________________________________
From: Utkarsh Gupta [[EMAIL PROTECTED]]
Sent: Wednesday, April 04, 2012 12:22 PM
To: [EMAIL PROTECTED]
Subject: Including third party jar files in Map Reduce job
Hi All,
I am new to Hadoop and was trying to generate random numbers using apache commons math library.
I used Netbeans to build the jar file and the manifest has path to commons-math jar as lib/commons-math3.jar
I have placed this jar file in HADOOP_HOME/lib folder but still I am getting ClassNotFoundException.
I tried using –libjar option with $HADOOP_HOME/bin/Hadoop jar myprg.jar <inputpath> <outputpath> –libjar <jarpath>
And $HADOOP_HOME/bin/Hadoop jar myprg.jar –libjar <jarpath> <inputpath> <outputpath>
But this is not working. Please help.
Thanks and Regards
Utkarsh Gupta
**************** CAUTION - Disclaimer *****************
This e-mail contains PRIVILEGED AND CONFIDENTIAL INFORMATION intended solely
for the use of the addressee(s). If you are not the intended recipient, please
notify the sender by e-mail and delete the original message. Further, you are not
to copy, disclose, or distribute this e-mail or its contents to any other person and
any such actions are unlawful. This e-mail may contain viruses. Infosys has taken
every reasonable precaution to minimize this risk, but is not liable for any damage
you may sustain as a result of any virus in this e-mail. You should carry out your
own virus checks before opening the e-mail or attachment. Infosys reserves the
right to monitor and review the content of all messages sent to or from this e-mail
address. Messages sent to or from this e-mail address may be stored on the
Infosys e-mail system.
***INFOSYS******** End of Disclaimer ********INFOSYS***
The usage of the jar command is like this,
Usage: hadoop jar <jar> [mainClass] args...
If you want the commons-math3.jar to be available for all the tasks you can do any one of these 1. Copy the jar file in $HADOOP_HOME/lib dir or 2. Use the generic option -libjars.
Can you give the stack trace of your problem for which class it is giving ClassNotFoundException(i.e for main class or math lib class)?
Thanks
Devaraj
________________________________________
From: Utkarsh Gupta [[EMAIL PROTECTED]]
Sent: Wednesday, April 04, 2012 12:22 PM
To: [EMAIL PROTECTED]
Subject: Including third party jar files in Map Reduce job
Hi All,
I am new to Hadoop and was trying to generate random numbers using apache commons math library.
I used Netbeans to build the jar file and the manifest has path to commons-math jar as lib/commons-math3.jar
I have placed this jar file in HADOOP_HOME/lib folder but still I am getting ClassNotFoundException.
I tried using –libjar option with $HADOOP_HOME/bin/Hadoop jar myprg.jar <inputpath> <outputpath> –libjar <jarpath>
And $HADOOP_HOME/bin/Hadoop jar myprg.jar –libjar <jarpath> <inputpath> <outputpath>
But this is not working. Please help.
Thanks and Regards
Utkarsh Gupta
**************** CAUTION - Disclaimer *****************
This e-mail contains PRIVILEGED AND CONFIDENTIAL INFORMATION intended solely
for the use of the addressee(s). If you are not the intended recipient, please
notify the sender by e-mail and delete the original message. Further, you are not
to copy, disclose, or distribute this e-mail or its contents to any other person and
any such actions are unlawful. This e-mail may contain viruses. Infosys has taken
every reasonable precaution to minimize this risk, but is not liable for any damage
you may sustain as a result of any virus in this e-mail. You should carry out your
own virus checks before opening the e-mail or attachment. Infosys reserves the
right to monitor and review the content of all messages sent to or from this e-mail
address. Messages sent to or from this e-mail address may be stored on the
Infosys e-mail system.
***INFOSYS******** End of Disclaimer ********INFOSYS***
-
RE: Including third party jar files in Map Reduce jobUtkarsh Gupta 2012-04-04, 07:25
Hi Devaraj,
I have already copied the required jar file in $HADOOP_HOME/lib folder.
Can you tell me where to add generic option -libjars
The stack trace is:
hadoop$ bin/hadoop jar WordCount.jar /user/hduser1/input/ /user/hduser1/output
12/04/04 12:45:51 WARN mapred.JobClient: Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same.
12/04/04 12:45:51 INFO input.FileInputFormat: Total input paths to process : 1
12/04/04 12:45:51 INFO mapred.JobClient: Running job: job_201204041107_0005
12/04/04 12:45:52 INFO mapred.JobClient: map 0% reduce 0%
12/04/04 12:46:07 INFO mapred.JobClient: Task Id : attempt_201204041107_0005_m_000000_0, Status : FAILED
Error: java.lang.ClassNotFoundException: org.apache.commons.math3.random.RandomDataImpl
at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
at wordcount.MyMapper.map(MyMapper.java:22)
at wordcount.MyMapper.map(MyMapper.java:14)
at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:763)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:369)
at org.apache.hadoop.mapred.Child$4.run(Child.java:259)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1059)
at org.apache.hadoop.mapred.Child.main(Child.java:253)
Thanks and Regards
Utkarsh
-----Original Message-----
From: Devaraj k [mailto:[EMAIL PROTECTED]]
Sent: Wednesday, April 04, 2012 12:35 PM
To: [EMAIL PROTECTED]
Subject: RE: Including third party jar files in Map Reduce job
Hi Utkarsh,
The usage of the jar command is like this,
Usage: hadoop jar <jar> [mainClass] args...
If you want the commons-math3.jar to be available for all the tasks you can do any one of these 1. Copy the jar file in $HADOOP_HOME/lib dir or 2. Use the generic option -libjars.
Can you give the stack trace of your problem for which class it is giving ClassNotFoundException(i.e for main class or math lib class)?
Thanks
Devaraj
________________________________________
From: Utkarsh Gupta [[EMAIL PROTECTED]]
Sent: Wednesday, April 04, 2012 12:22 PM
To: [EMAIL PROTECTED]
Subject: Including third party jar files in Map Reduce job
Hi All,
I am new to Hadoop and was trying to generate random numbers using apache commons math library.
I used Netbeans to build the jar file and the manifest has path to commons-math jar as lib/commons-math3.jar I have placed this jar file in HADOOP_HOME/lib folder but still I am getting ClassNotFoundException.
I tried using -libjars option with $HADOOP_HOME/bin/Hadoop jar myprg.jar <inputpath> <outputpath> -libjars <jarpath> And $HADOOP_HOME/bin/Hadoop jar myprg.jar -libjar <jarpath> <inputpath> <outputpath> But this is not working. Please help.
Thanks and Regards
Utkarsh Gupta
**************** CAUTION - Disclaimer ***************** This e-mail contains PRIVILEGED AND CONFIDENTIAL INFORMATION intended solely for the use of the addressee(s). If you are not the intended recipient, please notify the sender by e-mail and delete the original message. Further, you are not to copy, disclose, or distribute this e-mail or its contents to any other person and any such actions are unlawful. This e-mail may contain viruses. Infosys has taken every reasonable precaution to minimize this risk, but is not liable for any damage you may sustain as a result of any virus in this e-mail. You should carry out your own virus checks before opening the e-mail or attachment. Infosys reserves the right to monitor and review the content of all messages sent to or from this e-mail address. Messages sent to or from this e-mail address may be stored on the Infosys e-mail system.
***INFOSYS******** End of Disclaimer ********INFOSYS***
I have already copied the required jar file in $HADOOP_HOME/lib folder.
Can you tell me where to add generic option -libjars
The stack trace is:
hadoop$ bin/hadoop jar WordCount.jar /user/hduser1/input/ /user/hduser1/output
12/04/04 12:45:51 WARN mapred.JobClient: Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same.
12/04/04 12:45:51 INFO input.FileInputFormat: Total input paths to process : 1
12/04/04 12:45:51 INFO mapred.JobClient: Running job: job_201204041107_0005
12/04/04 12:45:52 INFO mapred.JobClient: map 0% reduce 0%
12/04/04 12:46:07 INFO mapred.JobClient: Task Id : attempt_201204041107_0005_m_000000_0, Status : FAILED
Error: java.lang.ClassNotFoundException: org.apache.commons.math3.random.RandomDataImpl
at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
at wordcount.MyMapper.map(MyMapper.java:22)
at wordcount.MyMapper.map(MyMapper.java:14)
at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:763)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:369)
at org.apache.hadoop.mapred.Child$4.run(Child.java:259)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1059)
at org.apache.hadoop.mapred.Child.main(Child.java:253)
Thanks and Regards
Utkarsh
-----Original Message-----
From: Devaraj k [mailto:[EMAIL PROTECTED]]
Sent: Wednesday, April 04, 2012 12:35 PM
To: [EMAIL PROTECTED]
Subject: RE: Including third party jar files in Map Reduce job
Hi Utkarsh,
The usage of the jar command is like this,
Usage: hadoop jar <jar> [mainClass] args...
If you want the commons-math3.jar to be available for all the tasks you can do any one of these 1. Copy the jar file in $HADOOP_HOME/lib dir or 2. Use the generic option -libjars.
Can you give the stack trace of your problem for which class it is giving ClassNotFoundException(i.e for main class or math lib class)?
Thanks
Devaraj
________________________________________
From: Utkarsh Gupta [[EMAIL PROTECTED]]
Sent: Wednesday, April 04, 2012 12:22 PM
To: [EMAIL PROTECTED]
Subject: Including third party jar files in Map Reduce job
Hi All,
I am new to Hadoop and was trying to generate random numbers using apache commons math library.
I used Netbeans to build the jar file and the manifest has path to commons-math jar as lib/commons-math3.jar I have placed this jar file in HADOOP_HOME/lib folder but still I am getting ClassNotFoundException.
I tried using -libjars option with $HADOOP_HOME/bin/Hadoop jar myprg.jar <inputpath> <outputpath> -libjars <jarpath> And $HADOOP_HOME/bin/Hadoop jar myprg.jar -libjar <jarpath> <inputpath> <outputpath> But this is not working. Please help.
Thanks and Regards
Utkarsh Gupta
**************** CAUTION - Disclaimer ***************** This e-mail contains PRIVILEGED AND CONFIDENTIAL INFORMATION intended solely for the use of the addressee(s). If you are not the intended recipient, please notify the sender by e-mail and delete the original message. Further, you are not to copy, disclose, or distribute this e-mail or its contents to any other person and any such actions are unlawful. This e-mail may contain viruses. Infosys has taken every reasonable precaution to minimize this risk, but is not liable for any damage you may sustain as a result of any virus in this e-mail. You should carry out your own virus checks before opening the e-mail or attachment. Infosys reserves the right to monitor and review the content of all messages sent to or from this e-mail address. Messages sent to or from this e-mail address may be stored on the Infosys e-mail system.
***INFOSYS******** End of Disclaimer ********INFOSYS***
-
Re: Including third party jar files in Map Reduce jobBejoy Ks 2012-04-04, 07:36
Hi Utkarsh
You can add third party jars to your map reduce job elegantly in
the following ways
1) use - libjars
hadoop jar jarName.jar com.driver.ClassName -libjars
/home/some/dir/somejar.jar ....
2) include the third pary jars in /lib folder while packaging your
application
3) If you are adding the jar in HADOOP_HOME/lib , you need to add this at
all nodes.
Regards
Bejoy KS
On Wed, Apr 4, 2012 at 12:55 PM, Utkarsh Gupta <[EMAIL PROTECTED]>wrote:
> Hi Devaraj,
>
> I have already copied the required jar file in $HADOOP_HOME/lib folder.
> Can you tell me where to add generic option -libjars
>
> The stack trace is:
> hadoop$ bin/hadoop jar WordCount.jar /user/hduser1/input/
> /user/hduser1/output
> 12/04/04 12:45:51 WARN mapred.JobClient: Use GenericOptionsParser for
> parsing the arguments. Applications should implement Tool for the same.
> 12/04/04 12:45:51 INFO input.FileInputFormat: Total input paths to process
> : 1
> 12/04/04 12:45:51 INFO mapred.JobClient: Running job: job_201204041107_0005
> 12/04/04 12:45:52 INFO mapred.JobClient: map 0% reduce 0%
> 12/04/04 12:46:07 INFO mapred.JobClient: Task Id :
> attempt_201204041107_0005_m_000000_0, Status : FAILED
> Error: java.lang.ClassNotFoundException:
> org.apache.commons.math3.random.RandomDataImpl
> at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
> at wordcount.MyMapper.map(MyMapper.java:22)
> at wordcount.MyMapper.map(MyMapper.java:14)
> at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:763)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:369)
> at org.apache.hadoop.mapred.Child$4.run(Child.java:259)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1059)
> at org.apache.hadoop.mapred.Child.main(Child.java:253)
>
> Thanks and Regards
> Utkarsh
>
> -----Original Message-----
> From: Devaraj k [mailto:[EMAIL PROTECTED]]
> Sent: Wednesday, April 04, 2012 12:35 PM
> To: [EMAIL PROTECTED]
> Subject: RE: Including third party jar files in Map Reduce job
>
> Hi Utkarsh,
>
> The usage of the jar command is like this,
>
> Usage: hadoop jar <jar> [mainClass] args...
>
> If you want the commons-math3.jar to be available for all the tasks you
> can do any one of these 1. Copy the jar file in $HADOOP_HOME/lib dir or 2.
> Use the generic option -libjars.
>
> Can you give the stack trace of your problem for which class it is giving
> ClassNotFoundException(i.e for main class or math lib class)?
>
> Thanks
> Devaraj
> ________________________________________
> From: Utkarsh Gupta [[EMAIL PROTECTED]]
> Sent: Wednesday, April 04, 2012 12:22 PM
> To: [EMAIL PROTECTED]
> Subject: Including third party jar files in Map Reduce job
>
> Hi All,
>
> I am new to Hadoop and was trying to generate random numbers using apache
> commons math library.
> I used Netbeans to build the jar file and the manifest has path to
> commons-math jar as lib/commons-math3.jar I have placed this jar file in
> HADOOP_HOME/lib folder but still I am getting ClassNotFoundException.
> I tried using -libjars option with $HADOOP_HOME/bin/Hadoop jar myprg.jar
> <inputpath> <outputpath> -libjars <jarpath> And $HADOOP_HOME/bin/Hadoop jar
> myprg.jar -libjar <jarpath> <inputpath> <outputpath> But this is not
> working. Please help.
>
>
> Thanks and Regards
> Utkarsh Gupta
>
>
>
> **************** CAUTION - Disclaimer ***************** This e-mail
You can add third party jars to your map reduce job elegantly in
the following ways
1) use - libjars
hadoop jar jarName.jar com.driver.ClassName -libjars
/home/some/dir/somejar.jar ....
2) include the third pary jars in /lib folder while packaging your
application
3) If you are adding the jar in HADOOP_HOME/lib , you need to add this at
all nodes.
Regards
Bejoy KS
On Wed, Apr 4, 2012 at 12:55 PM, Utkarsh Gupta <[EMAIL PROTECTED]>wrote:
> Hi Devaraj,
>
> I have already copied the required jar file in $HADOOP_HOME/lib folder.
> Can you tell me where to add generic option -libjars
>
> The stack trace is:
> hadoop$ bin/hadoop jar WordCount.jar /user/hduser1/input/
> /user/hduser1/output
> 12/04/04 12:45:51 WARN mapred.JobClient: Use GenericOptionsParser for
> parsing the arguments. Applications should implement Tool for the same.
> 12/04/04 12:45:51 INFO input.FileInputFormat: Total input paths to process
> : 1
> 12/04/04 12:45:51 INFO mapred.JobClient: Running job: job_201204041107_0005
> 12/04/04 12:45:52 INFO mapred.JobClient: map 0% reduce 0%
> 12/04/04 12:46:07 INFO mapred.JobClient: Task Id :
> attempt_201204041107_0005_m_000000_0, Status : FAILED
> Error: java.lang.ClassNotFoundException:
> org.apache.commons.math3.random.RandomDataImpl
> at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
> at wordcount.MyMapper.map(MyMapper.java:22)
> at wordcount.MyMapper.map(MyMapper.java:14)
> at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:763)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:369)
> at org.apache.hadoop.mapred.Child$4.run(Child.java:259)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1059)
> at org.apache.hadoop.mapred.Child.main(Child.java:253)
>
> Thanks and Regards
> Utkarsh
>
> -----Original Message-----
> From: Devaraj k [mailto:[EMAIL PROTECTED]]
> Sent: Wednesday, April 04, 2012 12:35 PM
> To: [EMAIL PROTECTED]
> Subject: RE: Including third party jar files in Map Reduce job
>
> Hi Utkarsh,
>
> The usage of the jar command is like this,
>
> Usage: hadoop jar <jar> [mainClass] args...
>
> If you want the commons-math3.jar to be available for all the tasks you
> can do any one of these 1. Copy the jar file in $HADOOP_HOME/lib dir or 2.
> Use the generic option -libjars.
>
> Can you give the stack trace of your problem for which class it is giving
> ClassNotFoundException(i.e for main class or math lib class)?
>
> Thanks
> Devaraj
> ________________________________________
> From: Utkarsh Gupta [[EMAIL PROTECTED]]
> Sent: Wednesday, April 04, 2012 12:22 PM
> To: [EMAIL PROTECTED]
> Subject: Including third party jar files in Map Reduce job
>
> Hi All,
>
> I am new to Hadoop and was trying to generate random numbers using apache
> commons math library.
> I used Netbeans to build the jar file and the manifest has path to
> commons-math jar as lib/commons-math3.jar I have placed this jar file in
> HADOOP_HOME/lib folder but still I am getting ClassNotFoundException.
> I tried using -libjars option with $HADOOP_HOME/bin/Hadoop jar myprg.jar
> <inputpath> <outputpath> -libjars <jarpath> And $HADOOP_HOME/bin/Hadoop jar
> myprg.jar -libjar <jarpath> <inputpath> <outputpath> But this is not
> working. Please help.
>
>
> Thanks and Regards
> Utkarsh Gupta
>
>
>
> **************** CAUTION - Disclaimer ***************** This e-mail
-
RE: Including third party jar files in Map Reduce jobDevaraj k 2012-04-04, 08:38
As Bejoy mentioned,
If you have copied the jar to $HADOOP_HOME, then you should copy it to all the nodes in the cluster. (or)
If you want to make use of -libjar option, your application should implement Tool to support generic options. Please check the below link for more details.
http://hadoop.apache.org/common/docs/current/commands_manual.html#jar
Thanks
Devaraj
________________________________________
From: Bejoy Ks [[EMAIL PROTECTED]]
Sent: Wednesday, April 04, 2012 1:06 PM
To: [EMAIL PROTECTED]
Subject: Re: Including third party jar files in Map Reduce job
Hi Utkarsh
You can add third party jars to your map reduce job elegantly in the following ways
1) use - libjars
hadoop jar jarName.jar com.driver.ClassName -libjars /home/some/dir/somejar.jar ....
2) include the third pary jars in /lib folder while packaging your application
3) If you are adding the jar in HADOOP_HOME/lib , you need to add this at all nodes.
Regards
Bejoy KS
On Wed, Apr 4, 2012 at 12:55 PM, Utkarsh Gupta <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>> wrote:
Hi Devaraj,
I have already copied the required jar file in $HADOOP_HOME/lib folder.
Can you tell me where to add generic option -libjars
The stack trace is:
hadoop$ bin/hadoop jar WordCount.jar /user/hduser1/input/ /user/hduser1/output
12/04/04 12:45:51 WARN mapred.JobClient: Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same.
12/04/04 12:45:51 INFO input.FileInputFormat: Total input paths to process : 1
12/04/04 12:45:51 INFO mapred.JobClient: Running job: job_201204041107_0005
12/04/04 12:45:52 INFO mapred.JobClient: map 0% reduce 0%
12/04/04 12:46:07 INFO mapred.JobClient: Task Id : attempt_201204041107_0005_m_000000_0, Status : FAILED
Error: java.lang.ClassNotFoundException: org.apache.commons.math3.random.RandomDataImpl
at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
at wordcount.MyMapper.map(MyMapper.java:22)
at wordcount.MyMapper.map(MyMapper.java:14)
at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:763)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:369)
at org.apache.hadoop.mapred.Child$4.run(Child.java:259)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1059)
at org.apache.hadoop.mapred.Child.main(Child.java:253)
Thanks and Regards
Utkarsh
-----Original Message-----
From: Devaraj k [mailto:[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>]
Sent: Wednesday, April 04, 2012 12:35 PM
To: [EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>
Subject: RE: Including third party jar files in Map Reduce job
Hi Utkarsh,
The usage of the jar command is like this,
Usage: hadoop jar <jar> [mainClass] args...
If you want the commons-math3.jar to be available for all the tasks you can do any one of these 1. Copy the jar file in $HADOOP_HOME/lib dir or 2. Use the generic option -libjars.
Can you give the stack trace of your problem for which class it is giving ClassNotFoundException(i.e for main class or math lib class)?
Thanks
Devaraj
________________________________________
From: Utkarsh Gupta [[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>]
Sent: Wednesday, April 04, 2012 12:22 PM
To: [EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>
Subject: Including third party jar files in Map Reduce job
Hi All,
I am new to Hadoop and was trying to generate random numbers using apache commons math library.
I used Netbeans to build the jar file and the manifest has path to commons-math jar as lib/commons-math3.jar I have placed this jar file in HADOOP_HOME/lib folder but still I am getting ClassNotFoundException.
I tried using -libjars option with $HADOOP_HOME/bin/Hadoop jar myprg.jar <inputpath> <outputpath> -libjars <jarpath> And $HADOOP_HOME/bin/Hadoop jar myprg.jar -libjar <jarpath> <inputpath> <outputpath> But this is not working. Please help.
Thanks and Regards
Utkarsh Gupta
**************** CAUTION - Disclaimer ***************** This e-mail contains PRIVILEGED AND CONFIDENTIAL INFORMATION intended solely for the use of the addressee(s). If you are not the intended recipient, please notify the sender by e-mail and delete the original message. Further, you are not to copy, disclose, or distribute this e-mail or its contents to any other person and any such actions are unlawful. This e-mail may contain viruses. Infosys has taken every reasonable precaution to minimize this risk, but is not liable for any damage you may sustain as a result of any virus in this e-mail. You should carry out your own virus checks before opening the e-mail or attachment. Infosys reserves the right to monitor and review the content of all messages sent to or from this e-mail address. Messages sent to or from this e-mail address may be stored on the Infosys e-mail system.
***INFOSYS******** End of Disclaimer ********INFOSYS***
If you have copied the jar to $HADOOP_HOME, then you should copy it to all the nodes in the cluster. (or)
If you want to make use of -libjar option, your application should implement Tool to support generic options. Please check the below link for more details.
http://hadoop.apache.org/common/docs/current/commands_manual.html#jar
Thanks
Devaraj
________________________________________
From: Bejoy Ks [[EMAIL PROTECTED]]
Sent: Wednesday, April 04, 2012 1:06 PM
To: [EMAIL PROTECTED]
Subject: Re: Including third party jar files in Map Reduce job
Hi Utkarsh
You can add third party jars to your map reduce job elegantly in the following ways
1) use - libjars
hadoop jar jarName.jar com.driver.ClassName -libjars /home/some/dir/somejar.jar ....
2) include the third pary jars in /lib folder while packaging your application
3) If you are adding the jar in HADOOP_HOME/lib , you need to add this at all nodes.
Regards
Bejoy KS
On Wed, Apr 4, 2012 at 12:55 PM, Utkarsh Gupta <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>> wrote:
Hi Devaraj,
I have already copied the required jar file in $HADOOP_HOME/lib folder.
Can you tell me where to add generic option -libjars
The stack trace is:
hadoop$ bin/hadoop jar WordCount.jar /user/hduser1/input/ /user/hduser1/output
12/04/04 12:45:51 WARN mapred.JobClient: Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same.
12/04/04 12:45:51 INFO input.FileInputFormat: Total input paths to process : 1
12/04/04 12:45:51 INFO mapred.JobClient: Running job: job_201204041107_0005
12/04/04 12:45:52 INFO mapred.JobClient: map 0% reduce 0%
12/04/04 12:46:07 INFO mapred.JobClient: Task Id : attempt_201204041107_0005_m_000000_0, Status : FAILED
Error: java.lang.ClassNotFoundException: org.apache.commons.math3.random.RandomDataImpl
at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
at wordcount.MyMapper.map(MyMapper.java:22)
at wordcount.MyMapper.map(MyMapper.java:14)
at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:763)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:369)
at org.apache.hadoop.mapred.Child$4.run(Child.java:259)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1059)
at org.apache.hadoop.mapred.Child.main(Child.java:253)
Thanks and Regards
Utkarsh
-----Original Message-----
From: Devaraj k [mailto:[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>]
Sent: Wednesday, April 04, 2012 12:35 PM
To: [EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>
Subject: RE: Including third party jar files in Map Reduce job
Hi Utkarsh,
The usage of the jar command is like this,
Usage: hadoop jar <jar> [mainClass] args...
If you want the commons-math3.jar to be available for all the tasks you can do any one of these 1. Copy the jar file in $HADOOP_HOME/lib dir or 2. Use the generic option -libjars.
Can you give the stack trace of your problem for which class it is giving ClassNotFoundException(i.e for main class or math lib class)?
Thanks
Devaraj
________________________________________
From: Utkarsh Gupta [[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>]
Sent: Wednesday, April 04, 2012 12:22 PM
To: [EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>
Subject: Including third party jar files in Map Reduce job
Hi All,
I am new to Hadoop and was trying to generate random numbers using apache commons math library.
I used Netbeans to build the jar file and the manifest has path to commons-math jar as lib/commons-math3.jar I have placed this jar file in HADOOP_HOME/lib folder but still I am getting ClassNotFoundException.
I tried using -libjars option with $HADOOP_HOME/bin/Hadoop jar myprg.jar <inputpath> <outputpath> -libjars <jarpath> And $HADOOP_HOME/bin/Hadoop jar myprg.jar -libjar <jarpath> <inputpath> <outputpath> But this is not working. Please help.
Thanks and Regards
Utkarsh Gupta
**************** CAUTION - Disclaimer ***************** This e-mail contains PRIVILEGED AND CONFIDENTIAL INFORMATION intended solely for the use of the addressee(s). If you are not the intended recipient, please notify the sender by e-mail and delete the original message. Further, you are not to copy, disclose, or distribute this e-mail or its contents to any other person and any such actions are unlawful. This e-mail may contain viruses. Infosys has taken every reasonable precaution to minimize this risk, but is not liable for any damage you may sustain as a result of any virus in this e-mail. You should carry out your own virus checks before opening the e-mail or attachment. Infosys reserves the right to monitor and review the content of all messages sent to or from this e-mail address. Messages sent to or from this e-mail address may be stored on the Infosys e-mail system.
***INFOSYS******** End of Disclaimer ********INFOSYS***
-
RE: Including third party jar files in Map Reduce jobUtkarsh Gupta 2012-04-04, 09:59
I have tried implementing Tool interface as mentioned @ http://hadoop.apache.org/common/docs/current/api/org/apache/hadoop/util/Tool.html
But -libjars option is not working.
I have copied the jar to all the nodes at $HADOOP_HOME/lib folder
But I am still getting the same error. The map task completes for master node (also acting as worker) from where job is run, but the task is failing on all the other nodes
The stack trace is :
hadoop$ bin/hadoop jar /home/hduser1/NetbeansProjects/WordCount/dist/WordCount.jar /user/hduser1/input/ /user/hduser1/output
12/04/04 15:21:00 WARN mapred.JobClient: Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same.
12/04/04 15:21:00 INFO input.FileInputFormat: Total input paths to process : 7
12/04/04 15:21:00 INFO mapred.JobClient: Running job: job_201204041107_0015
12/04/04 15:21:01 INFO mapred.JobClient: map 0% reduce 0%
12/04/04 15:21:17 INFO mapred.JobClient: map 27% reduce 0%
12/04/04 15:21:18 INFO mapred.JobClient: Task Id : attempt_201204041107_0015_m_000004_0, Status : FAILED
Error: java.lang.ClassNotFoundException: org.apache.commons.math3.random.RandomDataImpl
at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
at wordcount.MyMapper.map(MyMapper.java:22)
at wordcount.MyMapper.map(MyMapper.java:14)
at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:763)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:369)
at org.apache.hadoop.mapred.Child$4.run(Child.java:259)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1059)
at org.apache.hadoop.mapred.Child.main(Child.java:253)
Thanks and Regards
Utkarsh
-----Original Message-----
From: Devaraj k [mailto:[EMAIL PROTECTED]]
Sent: Wednesday, April 04, 2012 2:08 PM
To: [EMAIL PROTECTED]
Subject: RE: Including third party jar files in Map Reduce job
As Bejoy mentioned,
If you have copied the jar to $HADOOP_HOME, then you should copy it to all the nodes in the cluster. (or)
If you want to make use of -libjar option, your application should implement Tool to support generic options. Please check the below link for more details.
http://hadoop.apache.org/common/docs/current/commands_manual.html#jar
Thanks
Devaraj
________________________________________
From: Bejoy Ks [[EMAIL PROTECTED]]
Sent: Wednesday, April 04, 2012 1:06 PM
To: [EMAIL PROTECTED]
Subject: Re: Including third party jar files in Map Reduce job
Hi Utkarsh
You can add third party jars to your map reduce job elegantly in the following ways
1) use - libjars
hadoop jar jarName.jar com.driver.ClassName -libjars /home/some/dir/somejar.jar ....
2) include the third pary jars in /lib folder while packaging your application
3) If you are adding the jar in HADOOP_HOME/lib , you need to add this at all nodes.
Regards
Bejoy KS
On Wed, Apr 4, 2012 at 12:55 PM, Utkarsh Gupta <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>> wrote:
Hi Devaraj,
I have already copied the required jar file in $HADOOP_HOME/lib folder.
Can you tell me where to add generic option -libjars
The stack trace is:
hadoop$ bin/hadoop jar WordCount.jar /user/hduser1/input/ /user/hduser1/output
12/04/04 12:45:51 WARN mapred.JobClient: Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same.
12/04/04 12:45:51 INFO input.FileInputFormat: Total input paths to process : 1
12/04/04 12:45:51 INFO mapred.JobClient: Running job: job_201204041107_0005
12/04/04 12:45:52 INFO mapred.JobClient: map 0% reduce 0%
12/04/04 12:46:07 INFO mapred.JobClient: Task Id : attempt_201204041107_0005_m_000000_0, Status : FAILED
Error: java.lang.ClassNotFoundException: org.apache.commons.math3.random.RandomDataImpl
at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
at wordcount.MyMapper.map(MyMapper.java:22)
at wordcount.MyMapper.map(MyMapper.java:14)
at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:763)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:369)
at org.apache.hadoop.mapred.Child$4.run(Child.java:259)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1059)
at org.apache.hadoop.mapred.Child.main(Child.java:253)
Thanks and Regards
Utkarsh
From: Devaraj k [mailto:[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>]
Sent: Wednesday, April 04, 2012 12:35 PM
To: [EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>
Subject: RE: Including third party jar files in Map Reduce job
Hi Utkarsh,
The usage of the jar command is like this,
Usage: hadoop jar <jar> [mainClass] args...
If you want the commons-math3.jar to be available for all the tasks you can do any one of these 1. Copy the jar file in $HADOOP_HOME/lib dir or 2. Use the generic option -libjars.
Can you give the stack trace of your problem for which class it is giving ClassNotFoundException(i.e for mai
But -libjars option is not working.
I have copied the jar to all the nodes at $HADOOP_HOME/lib folder
But I am still getting the same error. The map task completes for master node (also acting as worker) from where job is run, but the task is failing on all the other nodes
The stack trace is :
hadoop$ bin/hadoop jar /home/hduser1/NetbeansProjects/WordCount/dist/WordCount.jar /user/hduser1/input/ /user/hduser1/output
12/04/04 15:21:00 WARN mapred.JobClient: Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same.
12/04/04 15:21:00 INFO input.FileInputFormat: Total input paths to process : 7
12/04/04 15:21:00 INFO mapred.JobClient: Running job: job_201204041107_0015
12/04/04 15:21:01 INFO mapred.JobClient: map 0% reduce 0%
12/04/04 15:21:17 INFO mapred.JobClient: map 27% reduce 0%
12/04/04 15:21:18 INFO mapred.JobClient: Task Id : attempt_201204041107_0015_m_000004_0, Status : FAILED
Error: java.lang.ClassNotFoundException: org.apache.commons.math3.random.RandomDataImpl
at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
at wordcount.MyMapper.map(MyMapper.java:22)
at wordcount.MyMapper.map(MyMapper.java:14)
at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:763)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:369)
at org.apache.hadoop.mapred.Child$4.run(Child.java:259)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1059)
at org.apache.hadoop.mapred.Child.main(Child.java:253)
Thanks and Regards
Utkarsh
-----Original Message-----
From: Devaraj k [mailto:[EMAIL PROTECTED]]
Sent: Wednesday, April 04, 2012 2:08 PM
To: [EMAIL PROTECTED]
Subject: RE: Including third party jar files in Map Reduce job
As Bejoy mentioned,
If you have copied the jar to $HADOOP_HOME, then you should copy it to all the nodes in the cluster. (or)
If you want to make use of -libjar option, your application should implement Tool to support generic options. Please check the below link for more details.
http://hadoop.apache.org/common/docs/current/commands_manual.html#jar
Thanks
Devaraj
________________________________________
From: Bejoy Ks [[EMAIL PROTECTED]]
Sent: Wednesday, April 04, 2012 1:06 PM
To: [EMAIL PROTECTED]
Subject: Re: Including third party jar files in Map Reduce job
Hi Utkarsh
You can add third party jars to your map reduce job elegantly in the following ways
1) use - libjars
hadoop jar jarName.jar com.driver.ClassName -libjars /home/some/dir/somejar.jar ....
2) include the third pary jars in /lib folder while packaging your application
3) If you are adding the jar in HADOOP_HOME/lib , you need to add this at all nodes.
Regards
Bejoy KS
On Wed, Apr 4, 2012 at 12:55 PM, Utkarsh Gupta <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>> wrote:
Hi Devaraj,
I have already copied the required jar file in $HADOOP_HOME/lib folder.
Can you tell me where to add generic option -libjars
The stack trace is:
hadoop$ bin/hadoop jar WordCount.jar /user/hduser1/input/ /user/hduser1/output
12/04/04 12:45:51 WARN mapred.JobClient: Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same.
12/04/04 12:45:51 INFO input.FileInputFormat: Total input paths to process : 1
12/04/04 12:45:51 INFO mapred.JobClient: Running job: job_201204041107_0005
12/04/04 12:45:52 INFO mapred.JobClient: map 0% reduce 0%
12/04/04 12:46:07 INFO mapred.JobClient: Task Id : attempt_201204041107_0005_m_000000_0, Status : FAILED
Error: java.lang.ClassNotFoundException: org.apache.commons.math3.random.RandomDataImpl
at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
at wordcount.MyMapper.map(MyMapper.java:22)
at wordcount.MyMapper.map(MyMapper.java:14)
at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:763)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:369)
at org.apache.hadoop.mapred.Child$4.run(Child.java:259)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1059)
at org.apache.hadoop.mapred.Child.main(Child.java:253)
Thanks and Regards
Utkarsh
From: Devaraj k [mailto:[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>]
Sent: Wednesday, April 04, 2012 12:35 PM
To: [EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>
Subject: RE: Including third party jar files in Map Reduce job
Hi Utkarsh,
The usage of the jar command is like this,
Usage: hadoop jar <jar> [mainClass] args...
If you want the commons-math3.jar to be available for all the tasks you can do any one of these 1. Copy the jar file in $HADOOP_HOME/lib dir or 2. Use the generic option -libjars.
Can you give the stack trace of your problem for which class it is giving ClassNotFoundException(i.e for mai
-
RE: Including third party jar files in Map Reduce jobUtkarsh Gupta 2012-04-04, 11:55
Hi Devaraj,
The code is running now after copying jar @ each node.
I might be doing some mistake previously.
Thanks Devaraj and Bejoy :)
-----Original Message-----
From: Devaraj k [mailto:[EMAIL PROTECTED]]
Sent: Wednesday, April 04, 2012 2:08 PM
To: [EMAIL PROTECTED]
Subject: RE: Including third party jar files in Map Reduce job
As Bejoy mentioned,
If you have copied the jar to $HADOOP_HOME, then you should copy it to all the nodes in the cluster. (or)
If you want to make use of -libjar option, your application should implement Tool to support generic options. Please check the below link for more details.
http://hadoop.apache.org/common/docs/current/commands_manual.html#jar
Thanks
Devaraj
________________________________________
From: Bejoy Ks [[EMAIL PROTECTED]]
Sent: Wednesday, April 04, 2012 1:06 PM
To: [EMAIL PROTECTED]
Subject: Re: Including third party jar files in Map Reduce job
Hi Utkarsh
You can add third party jars to your map reduce job elegantly in the following ways
1) use - libjars
hadoop jar jarName.jar com.driver.ClassName -libjars /home/some/dir/somejar.jar ....
2) include the third pary jars in /lib folder while packaging your application
3) If you are adding the jar in HADOOP_HOME/lib , you need to add this at all nodes.
Regards
Bejoy KS
On Wed, Apr 4, 2012 at 12:55 PM, Utkarsh Gupta <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>> wrote:
Hi Devaraj,
I have already copied the required jar file in $HADOOP_HOME/lib folder.
Can you tell me where to add generic option -libjars
The stack trace is:
hadoop$ bin/hadoop jar WordCount.jar /user/hduser1/input/ /user/hduser1/output
12/04/04 12:45:51 WARN mapred.JobClient: Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same.
12/04/04 12:45:51 INFO input.FileInputFormat: Total input paths to process : 1
12/04/04 12:45:51 INFO mapred.JobClient: Running job: job_201204041107_0005
12/04/04 12:45:52 INFO mapred.JobClient: map 0% reduce 0%
12/04/04 12:46:07 INFO mapred.JobClient: Task Id : attempt_201204041107_0005_m_000000_0, Status : FAILED
Error: java.lang.ClassNotFoundException: org.apache.commons.math3.random.RandomDataImpl
at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
at wordcount.MyMapper.map(MyMapper.java:22)
at wordcount.MyMapper.map(MyMapper.java:14)
at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:763)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:369)
at org.apache.hadoop.mapred.Child$4.run(Child.java:259)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1059)
at org.apache.hadoop.mapred.Child.main(Child.java:253)
Thanks and Regards
Utkarsh
-----Original Message-----
From: Devaraj k [mailto:[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>]
Sent: Wednesday, April 04, 2012 12:35 PM
To: [EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>
Subject: RE: Including third party jar files in Map Reduce job
Hi Utkarsh,
The usage of the jar command is like this,
Usage: hadoop jar <jar> [mainClass] args...
If you want the commons-math3.jar to be available for all the tasks you can do any one of these 1. Copy the jar file in $HADOOP_HOME/lib dir or 2. Use the generic option -libjars.
Can you give the stack trace of your problem for which class it is giving ClassNotFoundException(i.e for main class or math lib class)?
Thanks
Devaraj
________________________________________
From: Utkarsh Gupta [[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>]
Sent: Wednesday, April 04, 2012 12:22 PM
To: [EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>
Subject: Including third party jar files in Map Reduce job
Hi All,
I am new to Hadoop and was trying to generate random numbers using apache commons math library.
I used Netbeans to build the jar file and the manifest has path to commons-math jar as lib/commons-math3.jar I have placed this jar file in HADOOP_HOME/lib folder but still I am getting ClassNotFoundException.
I tried using -libjars option with $HADOOP_HOME/bin/Hadoop jar myprg.jar <inputpath> <outputpath> -libjars <jarpath> And $HADOOP_HOME/bin/Hadoop jar myprg.jar -libjar <jarpath> <inputpath> <outputpath> But this is not working. Please help.
Thanks and Regards
Utkarsh Gupta
**************** CAUTION - Disclaimer ***************** This e-mail contains PRIVILEGED AND CONFIDENTIAL INFORMATION intended solely for the use of the addressee(s). If you are not the intended recipient, please notify the sender by e-mail and delete the original message. Further, you are not to copy, disclose, or distribute this e-mail or its contents to any other person and any such actions are unlawful. This e-mail may contain viruses. Infosys has taken every reasonable precaution to minimize this risk, but is not liable for any damage you may sustain as a result of any virus in this e-mail. You should carry out your own virus checks before opening the e-mail or attachment. Infosys reserves the right to monitor and review the content of all messages sent to or from this e-mail address. Messages sent to or from this e-mail address may be stored on the Infosys e-mail system.
***INFOSYS******** End of Disclaimer ********INFOSYS***
The code is running now after copying jar @ each node.
I might be doing some mistake previously.
Thanks Devaraj and Bejoy :)
-----Original Message-----
From: Devaraj k [mailto:[EMAIL PROTECTED]]
Sent: Wednesday, April 04, 2012 2:08 PM
To: [EMAIL PROTECTED]
Subject: RE: Including third party jar files in Map Reduce job
As Bejoy mentioned,
If you have copied the jar to $HADOOP_HOME, then you should copy it to all the nodes in the cluster. (or)
If you want to make use of -libjar option, your application should implement Tool to support generic options. Please check the below link for more details.
http://hadoop.apache.org/common/docs/current/commands_manual.html#jar
Thanks
Devaraj
________________________________________
From: Bejoy Ks [[EMAIL PROTECTED]]
Sent: Wednesday, April 04, 2012 1:06 PM
To: [EMAIL PROTECTED]
Subject: Re: Including third party jar files in Map Reduce job
Hi Utkarsh
You can add third party jars to your map reduce job elegantly in the following ways
1) use - libjars
hadoop jar jarName.jar com.driver.ClassName -libjars /home/some/dir/somejar.jar ....
2) include the third pary jars in /lib folder while packaging your application
3) If you are adding the jar in HADOOP_HOME/lib , you need to add this at all nodes.
Regards
Bejoy KS
On Wed, Apr 4, 2012 at 12:55 PM, Utkarsh Gupta <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>> wrote:
Hi Devaraj,
I have already copied the required jar file in $HADOOP_HOME/lib folder.
Can you tell me where to add generic option -libjars
The stack trace is:
hadoop$ bin/hadoop jar WordCount.jar /user/hduser1/input/ /user/hduser1/output
12/04/04 12:45:51 WARN mapred.JobClient: Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same.
12/04/04 12:45:51 INFO input.FileInputFormat: Total input paths to process : 1
12/04/04 12:45:51 INFO mapred.JobClient: Running job: job_201204041107_0005
12/04/04 12:45:52 INFO mapred.JobClient: map 0% reduce 0%
12/04/04 12:46:07 INFO mapred.JobClient: Task Id : attempt_201204041107_0005_m_000000_0, Status : FAILED
Error: java.lang.ClassNotFoundException: org.apache.commons.math3.random.RandomDataImpl
at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
at wordcount.MyMapper.map(MyMapper.java:22)
at wordcount.MyMapper.map(MyMapper.java:14)
at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:763)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:369)
at org.apache.hadoop.mapred.Child$4.run(Child.java:259)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1059)
at org.apache.hadoop.mapred.Child.main(Child.java:253)
Thanks and Regards
Utkarsh
-----Original Message-----
From: Devaraj k [mailto:[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>]
Sent: Wednesday, April 04, 2012 12:35 PM
To: [EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>
Subject: RE: Including third party jar files in Map Reduce job
Hi Utkarsh,
The usage of the jar command is like this,
Usage: hadoop jar <jar> [mainClass] args...
If you want the commons-math3.jar to be available for all the tasks you can do any one of these 1. Copy the jar file in $HADOOP_HOME/lib dir or 2. Use the generic option -libjars.
Can you give the stack trace of your problem for which class it is giving ClassNotFoundException(i.e for main class or math lib class)?
Thanks
Devaraj
________________________________________
From: Utkarsh Gupta [[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>]
Sent: Wednesday, April 04, 2012 12:22 PM
To: [EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>
Subject: Including third party jar files in Map Reduce job
Hi All,
I am new to Hadoop and was trying to generate random numbers using apache commons math library.
I used Netbeans to build the jar file and the manifest has path to commons-math jar as lib/commons-math3.jar I have placed this jar file in HADOOP_HOME/lib folder but still I am getting ClassNotFoundException.
I tried using -libjars option with $HADOOP_HOME/bin/Hadoop jar myprg.jar <inputpath> <outputpath> -libjars <jarpath> And $HADOOP_HOME/bin/Hadoop jar myprg.jar -libjar <jarpath> <inputpath> <outputpath> But this is not working. Please help.
Thanks and Regards
Utkarsh Gupta
**************** CAUTION - Disclaimer ***************** This e-mail contains PRIVILEGED AND CONFIDENTIAL INFORMATION intended solely for the use of the addressee(s). If you are not the intended recipient, please notify the sender by e-mail and delete the original message. Further, you are not to copy, disclose, or distribute this e-mail or its contents to any other person and any such actions are unlawful. This e-mail may contain viruses. Infosys has taken every reasonable precaution to minimize this risk, but is not liable for any damage you may sustain as a result of any virus in this e-mail. You should carry out your own virus checks before opening the e-mail or attachment. Infosys reserves the right to monitor and review the content of all messages sent to or from this e-mail address. Messages sent to or from this e-mail address may be stored on the Infosys e-mail system.
***INFOSYS******** End of Disclaimer ********INFOSYS***
-
Re: Including third party jar files in Map Reduce jobHarsh J 2012-04-04, 12:49
Utkarsh,
A log like "12/04/04 15:21:00 WARN mapred.JobClient: Use
GenericOptionsParser for parsing the arguments. Applications should
implement Tool for the same." indicates you haven't implemented the
Tool approach properly (or aren't calling its run()).
On Wed, Apr 4, 2012 at 5:25 PM, Utkarsh Gupta <[EMAIL PROTECTED]> wrote:
> Hi Devaraj,
>
> The code is running now after copying jar @ each node.
> I might be doing some mistake previously.
> Thanks Devaraj and Bejoy :)
>
>
> -----Original Message-----
> From: Devaraj k [mailto:[EMAIL PROTECTED]]
> Sent: Wednesday, April 04, 2012 2:08 PM
> To: [EMAIL PROTECTED]
> Subject: RE: Including third party jar files in Map Reduce job
>
> As Bejoy mentioned,
>
> If you have copied the jar to $HADOOP_HOME, then you should copy it to all the nodes in the cluster. (or)
>
> If you want to make use of -libjar option, your application should implement Tool to support generic options. Please check the below link for more details.
>
> http://hadoop.apache.org/common/docs/current/commands_manual.html#jar
>
> Thanks
> Devaraj
> ________________________________________
> From: Bejoy Ks [[EMAIL PROTECTED]]
> Sent: Wednesday, April 04, 2012 1:06 PM
> To: [EMAIL PROTECTED]
> Subject: Re: Including third party jar files in Map Reduce job
>
> Hi Utkarsh
> You can add third party jars to your map reduce job elegantly in the following ways
>
> 1) use - libjars
> hadoop jar jarName.jar com.driver.ClassName -libjars /home/some/dir/somejar.jar ....
>
> 2) include the third pary jars in /lib folder while packaging your application
>
> 3) If you are adding the jar in HADOOP_HOME/lib , you need to add this at all nodes.
>
> Regards
> Bejoy KS
>
> On Wed, Apr 4, 2012 at 12:55 PM, Utkarsh Gupta <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>> wrote:
> Hi Devaraj,
>
> I have already copied the required jar file in $HADOOP_HOME/lib folder.
> Can you tell me where to add generic option -libjars
>
> The stack trace is:
> hadoop$ bin/hadoop jar WordCount.jar /user/hduser1/input/ /user/hduser1/output
> 12/04/04 12:45:51 WARN mapred.JobClient: Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same.
> 12/04/04 12:45:51 INFO input.FileInputFormat: Total input paths to process : 1
> 12/04/04 12:45:51 INFO mapred.JobClient: Running job: job_201204041107_0005
> 12/04/04 12:45:52 INFO mapred.JobClient: map 0% reduce 0%
> 12/04/04 12:46:07 INFO mapred.JobClient: Task Id : attempt_201204041107_0005_m_000000_0, Status : FAILED
> Error: java.lang.ClassNotFoundException: org.apache.commons.math3.random.RandomDataImpl
> at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
> at wordcount.MyMapper.map(MyMapper.java:22)
> at wordcount.MyMapper.map(MyMapper.java:14)
> at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:763)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:369)
> at org.apache.hadoop.mapred.Child$4.run(Child.java:259)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1059)
> at org.apache.hadoop.mapred.Child.main(Child.java:253)
>
> Thanks and Regards
> Utkarsh
>
> -----Original Message-----
> From: Devaraj k [mailto:[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>]
> Sent: Wednesday, April 04, 2012 12:35 PM
> To: [EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>
Harsh J
A log like "12/04/04 15:21:00 WARN mapred.JobClient: Use
GenericOptionsParser for parsing the arguments. Applications should
implement Tool for the same." indicates you haven't implemented the
Tool approach properly (or aren't calling its run()).
On Wed, Apr 4, 2012 at 5:25 PM, Utkarsh Gupta <[EMAIL PROTECTED]> wrote:
> Hi Devaraj,
>
> The code is running now after copying jar @ each node.
> I might be doing some mistake previously.
> Thanks Devaraj and Bejoy :)
>
>
> -----Original Message-----
> From: Devaraj k [mailto:[EMAIL PROTECTED]]
> Sent: Wednesday, April 04, 2012 2:08 PM
> To: [EMAIL PROTECTED]
> Subject: RE: Including third party jar files in Map Reduce job
>
> As Bejoy mentioned,
>
> If you have copied the jar to $HADOOP_HOME, then you should copy it to all the nodes in the cluster. (or)
>
> If you want to make use of -libjar option, your application should implement Tool to support generic options. Please check the below link for more details.
>
> http://hadoop.apache.org/common/docs/current/commands_manual.html#jar
>
> Thanks
> Devaraj
> ________________________________________
> From: Bejoy Ks [[EMAIL PROTECTED]]
> Sent: Wednesday, April 04, 2012 1:06 PM
> To: [EMAIL PROTECTED]
> Subject: Re: Including third party jar files in Map Reduce job
>
> Hi Utkarsh
> You can add third party jars to your map reduce job elegantly in the following ways
>
> 1) use - libjars
> hadoop jar jarName.jar com.driver.ClassName -libjars /home/some/dir/somejar.jar ....
>
> 2) include the third pary jars in /lib folder while packaging your application
>
> 3) If you are adding the jar in HADOOP_HOME/lib , you need to add this at all nodes.
>
> Regards
> Bejoy KS
>
> On Wed, Apr 4, 2012 at 12:55 PM, Utkarsh Gupta <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>> wrote:
> Hi Devaraj,
>
> I have already copied the required jar file in $HADOOP_HOME/lib folder.
> Can you tell me where to add generic option -libjars
>
> The stack trace is:
> hadoop$ bin/hadoop jar WordCount.jar /user/hduser1/input/ /user/hduser1/output
> 12/04/04 12:45:51 WARN mapred.JobClient: Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same.
> 12/04/04 12:45:51 INFO input.FileInputFormat: Total input paths to process : 1
> 12/04/04 12:45:51 INFO mapred.JobClient: Running job: job_201204041107_0005
> 12/04/04 12:45:52 INFO mapred.JobClient: map 0% reduce 0%
> 12/04/04 12:46:07 INFO mapred.JobClient: Task Id : attempt_201204041107_0005_m_000000_0, Status : FAILED
> Error: java.lang.ClassNotFoundException: org.apache.commons.math3.random.RandomDataImpl
> at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
> at wordcount.MyMapper.map(MyMapper.java:22)
> at wordcount.MyMapper.map(MyMapper.java:14)
> at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:763)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:369)
> at org.apache.hadoop.mapred.Child$4.run(Child.java:259)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1059)
> at org.apache.hadoop.mapred.Child.main(Child.java:253)
>
> Thanks and Regards
> Utkarsh
>
> -----Original Message-----
> From: Devaraj k [mailto:[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>]
> Sent: Wednesday, April 04, 2012 12:35 PM
> To: [EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>
Harsh J
-
RE: Including third party jar files in Map Reduce jobUtkarsh Gupta 2012-04-04, 12:55
Hi Harsh,
I have implemented Tool like this
public static void main(String[] args) throws Exception {
Configuration configuration = new Configuration();
int rc = ToolRunner.run(configuration, new WordCount(), args);
System.exit(rc);
}
@Override
public int run(String[] args) throws Exception {
if (args.length < 2) {
System.err.println("Usage: WordCount <input path> <output path>");
return -1;
}
Configuration conf = new Configuration();
//conf.set("mapred.job.tracker", "local");
Job job = new Job(conf, "wordcount");
job.setJarByClass(WordCount.class);
job.setMapperClass(MyMapper.class);
job.setReducerClass(MyReducer.class);
job.setOutputKeyClass(Text.class);
job.setOutputValueClass(IntWritable.class);
job.setInputFormatClass(TextInputFormat.class);
job.setOutputFormatClass(TextOutputFormat.class);
job.setNumReduceTasks(1);
FileInputFormat.addInputPath(job, new Path(args[0]));
FileOutputFormat.setOutputPath(job, new Path(args[1]));
return (job.waitForCompletion(true)) ? 0 : 1;
}
This is working but I am unable to figure out why still it is warning
-----Original Message-----
From: Harsh J [mailto:[EMAIL PROTECTED]]
Sent: Wednesday, April 04, 2012 6:20 PM
To: [EMAIL PROTECTED]
Subject: Re: Including third party jar files in Map Reduce job
Utkarsh,
A log like "12/04/04 15:21:00 WARN mapred.JobClient: Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same." indicates you haven't implemented the Tool approach properly (or aren't calling its run()).
On Wed, Apr 4, 2012 at 5:25 PM, Utkarsh Gupta <[EMAIL PROTECTED]> wrote:
> Hi Devaraj,
>
> The code is running now after copying jar @ each node.
> I might be doing some mistake previously.
> Thanks Devaraj and Bejoy :)
>
>
> -----Original Message-----
> From: Devaraj k [mailto:[EMAIL PROTECTED]]
> Sent: Wednesday, April 04, 2012 2:08 PM
> To: [EMAIL PROTECTED]
> Subject: RE: Including third party jar files in Map Reduce job
>
> As Bejoy mentioned,
>
> If you have copied the jar to $HADOOP_HOME, then you should copy it to
> all the nodes in the cluster. (or)
>
> If you want to make use of -libjar option, your application should implement Tool to support generic options. Please check the below link for more details.
>
> http://hadoop.apache.org/common/docs/current/commands_manual.html#jar
>
> Thanks
> Devaraj
> ________________________________________
> From: Bejoy Ks [[EMAIL PROTECTED]]
> Sent: Wednesday, April 04, 2012 1:06 PM
> To: [EMAIL PROTECTED]
> Subject: Re: Including third party jar files in Map Reduce job
>
> Hi Utkarsh
> You can add third party jars to your map reduce job elegantly
> in the following ways
>
> 1) use - libjars
> hadoop jar jarName.jar com.driver.ClassName -libjars /home/some/dir/somejar.jar ....
>
> 2) include the third pary jars in /lib folder while packaging your
> application
>
> 3) If you are adding the jar in HADOOP_HOME/lib , you need to add this at all nodes.
>
> Regards
> Bejoy KS
>
> On Wed, Apr 4, 2012 at 12:55 PM, Utkarsh Gupta <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>> wrote:
> Hi Devaraj,
>
> I have already copied the required jar file in $HADOOP_HOME/lib folder.
> Can you tell me where to add generic option -libjars
>
> The stack trace is:
> hadoop$ bin/hadoop jar WordCount.jar /user/hduser1/input/
> /user/hduser1/output
> 12/04/04 12:45:51 WARN mapred.JobClient: Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same.
> 12/04/04 12:45:51 INFO input.FileInputFormat: Total input paths to
> process : 1
> 12/04/04 12:45:51 INFO mapred.JobClient: Running job:
> job_201204041107_0005
> 12/04/04 12:45:52 INFO mapred.JobClient: map 0% reduce 0%
> 12/04/04 12:46:07 INFO mapred.JobClient: Task Id :
Harsh J
I have implemented Tool like this
public static void main(String[] args) throws Exception {
Configuration configuration = new Configuration();
int rc = ToolRunner.run(configuration, new WordCount(), args);
System.exit(rc);
}
@Override
public int run(String[] args) throws Exception {
if (args.length < 2) {
System.err.println("Usage: WordCount <input path> <output path>");
return -1;
}
Configuration conf = new Configuration();
//conf.set("mapred.job.tracker", "local");
Job job = new Job(conf, "wordcount");
job.setJarByClass(WordCount.class);
job.setMapperClass(MyMapper.class);
job.setReducerClass(MyReducer.class);
job.setOutputKeyClass(Text.class);
job.setOutputValueClass(IntWritable.class);
job.setInputFormatClass(TextInputFormat.class);
job.setOutputFormatClass(TextOutputFormat.class);
job.setNumReduceTasks(1);
FileInputFormat.addInputPath(job, new Path(args[0]));
FileOutputFormat.setOutputPath(job, new Path(args[1]));
return (job.waitForCompletion(true)) ? 0 : 1;
}
This is working but I am unable to figure out why still it is warning
-----Original Message-----
From: Harsh J [mailto:[EMAIL PROTECTED]]
Sent: Wednesday, April 04, 2012 6:20 PM
To: [EMAIL PROTECTED]
Subject: Re: Including third party jar files in Map Reduce job
Utkarsh,
A log like "12/04/04 15:21:00 WARN mapred.JobClient: Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same." indicates you haven't implemented the Tool approach properly (or aren't calling its run()).
On Wed, Apr 4, 2012 at 5:25 PM, Utkarsh Gupta <[EMAIL PROTECTED]> wrote:
> Hi Devaraj,
>
> The code is running now after copying jar @ each node.
> I might be doing some mistake previously.
> Thanks Devaraj and Bejoy :)
>
>
> -----Original Message-----
> From: Devaraj k [mailto:[EMAIL PROTECTED]]
> Sent: Wednesday, April 04, 2012 2:08 PM
> To: [EMAIL PROTECTED]
> Subject: RE: Including third party jar files in Map Reduce job
>
> As Bejoy mentioned,
>
> If you have copied the jar to $HADOOP_HOME, then you should copy it to
> all the nodes in the cluster. (or)
>
> If you want to make use of -libjar option, your application should implement Tool to support generic options. Please check the below link for more details.
>
> http://hadoop.apache.org/common/docs/current/commands_manual.html#jar
>
> Thanks
> Devaraj
> ________________________________________
> From: Bejoy Ks [[EMAIL PROTECTED]]
> Sent: Wednesday, April 04, 2012 1:06 PM
> To: [EMAIL PROTECTED]
> Subject: Re: Including third party jar files in Map Reduce job
>
> Hi Utkarsh
> You can add third party jars to your map reduce job elegantly
> in the following ways
>
> 1) use - libjars
> hadoop jar jarName.jar com.driver.ClassName -libjars /home/some/dir/somejar.jar ....
>
> 2) include the third pary jars in /lib folder while packaging your
> application
>
> 3) If you are adding the jar in HADOOP_HOME/lib , you need to add this at all nodes.
>
> Regards
> Bejoy KS
>
> On Wed, Apr 4, 2012 at 12:55 PM, Utkarsh Gupta <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>> wrote:
> Hi Devaraj,
>
> I have already copied the required jar file in $HADOOP_HOME/lib folder.
> Can you tell me where to add generic option -libjars
>
> The stack trace is:
> hadoop$ bin/hadoop jar WordCount.jar /user/hduser1/input/
> /user/hduser1/output
> 12/04/04 12:45:51 WARN mapred.JobClient: Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same.
> 12/04/04 12:45:51 INFO input.FileInputFormat: Total input paths to
> process : 1
> 12/04/04 12:45:51 INFO mapred.JobClient: Running job:
> job_201204041107_0005
> 12/04/04 12:45:52 INFO mapred.JobClient: map 0% reduce 0%
> 12/04/04 12:46:07 INFO mapred.JobClient: Task Id :
Harsh J
-
Re: Including third party jar files in Map Reduce jobHarsh J 2012-04-04, 13:01
When using Tool, do not use:
Configuration conf = new Configuration();
Instead get config from the class:
Configuration conf = getConf();
This is documented at
http://hadoop.apache.org/common/docs/current/api/org/apache/hadoop/util/Tool.html
On Wed, Apr 4, 2012 at 6:25 PM, Utkarsh Gupta <[EMAIL PROTECTED]> wrote:
> Hi Harsh,
> I have implemented Tool like this
>
> public static void main(String[] args) throws Exception {
> Configuration configuration = new Configuration();
> int rc = ToolRunner.run(configuration, new WordCount(), args);
> System.exit(rc);
> }
>
> @Override
> public int run(String[] args) throws Exception {
> if (args.length < 2) {
> System.err.println("Usage: WordCount <input path> <output path>");
> return -1;
> }
> Configuration conf = new Configuration();
> //conf.set("mapred.job.tracker", "local");
> Job job = new Job(conf, "wordcount");
>
> job.setJarByClass(WordCount.class);
> job.setMapperClass(MyMapper.class);
> job.setReducerClass(MyReducer.class);
> job.setOutputKeyClass(Text.class);
> job.setOutputValueClass(IntWritable.class);
> job.setInputFormatClass(TextInputFormat.class);
> job.setOutputFormatClass(TextOutputFormat.class);
> job.setNumReduceTasks(1);
> FileInputFormat.addInputPath(job, new Path(args[0]));
> FileOutputFormat.setOutputPath(job, new Path(args[1]));
> return (job.waitForCompletion(true)) ? 0 : 1;
> }
>
> This is working but I am unable to figure out why still it is warning
> -----Original Message-----
> From: Harsh J [mailto:[EMAIL PROTECTED]]
> Sent: Wednesday, April 04, 2012 6:20 PM
> To: [EMAIL PROTECTED]
> Subject: Re: Including third party jar files in Map Reduce job
>
> Utkarsh,
>
> A log like "12/04/04 15:21:00 WARN mapred.JobClient: Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same." indicates you haven't implemented the Tool approach properly (or aren't calling its run()).
>
> On Wed, Apr 4, 2012 at 5:25 PM, Utkarsh Gupta <[EMAIL PROTECTED]> wrote:
>> Hi Devaraj,
>>
>> The code is running now after copying jar @ each node.
>> I might be doing some mistake previously.
>> Thanks Devaraj and Bejoy :)
>>
>>
>> -----Original Message-----
>> From: Devaraj k [mailto:[EMAIL PROTECTED]]
>> Sent: Wednesday, April 04, 2012 2:08 PM
>> To: [EMAIL PROTECTED]
>> Subject: RE: Including third party jar files in Map Reduce job
>>
>> As Bejoy mentioned,
>>
>> If you have copied the jar to $HADOOP_HOME, then you should copy it to
>> all the nodes in the cluster. (or)
>>
>> If you want to make use of -libjar option, your application should implement Tool to support generic options. Please check the below link for more details.
>>
>> http://hadoop.apache.org/common/docs/current/commands_manual.html#jar
>>
>> Thanks
>> Devaraj
>> ________________________________________
>> From: Bejoy Ks [[EMAIL PROTECTED]]
>> Sent: Wednesday, April 04, 2012 1:06 PM
>> To: [EMAIL PROTECTED]
>> Subject: Re: Including third party jar files in Map Reduce job
>>
>> Hi Utkarsh
>> You can add third party jars to your map reduce job elegantly
>> in the following ways
>>
>> 1) use - libjars
>> hadoop jar jarName.jar com.driver.ClassName -libjars /home/some/dir/somejar.jar ....
>>
>> 2) include the third pary jars in /lib folder while packaging your
>> application
>>
>> 3) If you are adding the jar in HADOOP_HOME/lib , you need to add this at all nodes.
>>
>> Regards
>> Bejoy KS
>>
>> On Wed, Apr 4, 2012 at 12:55 PM, Utkarsh Gupta <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>> wrote:
>> Hi Devaraj,
>>
>> I have already copied the required jar file in $HADOOP_HOME/lib folder.
>> Can you tell me where to add generic option -libjars
>>
>> The stack trace is:
>> hadoop$ bin/hadoop jar WordCount.jar /user/hduser1/input/
Harsh J
Configuration conf = new Configuration();
Instead get config from the class:
Configuration conf = getConf();
This is documented at
http://hadoop.apache.org/common/docs/current/api/org/apache/hadoop/util/Tool.html
On Wed, Apr 4, 2012 at 6:25 PM, Utkarsh Gupta <[EMAIL PROTECTED]> wrote:
> Hi Harsh,
> I have implemented Tool like this
>
> public static void main(String[] args) throws Exception {
> Configuration configuration = new Configuration();
> int rc = ToolRunner.run(configuration, new WordCount(), args);
> System.exit(rc);
> }
>
> @Override
> public int run(String[] args) throws Exception {
> if (args.length < 2) {
> System.err.println("Usage: WordCount <input path> <output path>");
> return -1;
> }
> Configuration conf = new Configuration();
> //conf.set("mapred.job.tracker", "local");
> Job job = new Job(conf, "wordcount");
>
> job.setJarByClass(WordCount.class);
> job.setMapperClass(MyMapper.class);
> job.setReducerClass(MyReducer.class);
> job.setOutputKeyClass(Text.class);
> job.setOutputValueClass(IntWritable.class);
> job.setInputFormatClass(TextInputFormat.class);
> job.setOutputFormatClass(TextOutputFormat.class);
> job.setNumReduceTasks(1);
> FileInputFormat.addInputPath(job, new Path(args[0]));
> FileOutputFormat.setOutputPath(job, new Path(args[1]));
> return (job.waitForCompletion(true)) ? 0 : 1;
> }
>
> This is working but I am unable to figure out why still it is warning
> -----Original Message-----
> From: Harsh J [mailto:[EMAIL PROTECTED]]
> Sent: Wednesday, April 04, 2012 6:20 PM
> To: [EMAIL PROTECTED]
> Subject: Re: Including third party jar files in Map Reduce job
>
> Utkarsh,
>
> A log like "12/04/04 15:21:00 WARN mapred.JobClient: Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same." indicates you haven't implemented the Tool approach properly (or aren't calling its run()).
>
> On Wed, Apr 4, 2012 at 5:25 PM, Utkarsh Gupta <[EMAIL PROTECTED]> wrote:
>> Hi Devaraj,
>>
>> The code is running now after copying jar @ each node.
>> I might be doing some mistake previously.
>> Thanks Devaraj and Bejoy :)
>>
>>
>> -----Original Message-----
>> From: Devaraj k [mailto:[EMAIL PROTECTED]]
>> Sent: Wednesday, April 04, 2012 2:08 PM
>> To: [EMAIL PROTECTED]
>> Subject: RE: Including third party jar files in Map Reduce job
>>
>> As Bejoy mentioned,
>>
>> If you have copied the jar to $HADOOP_HOME, then you should copy it to
>> all the nodes in the cluster. (or)
>>
>> If you want to make use of -libjar option, your application should implement Tool to support generic options. Please check the below link for more details.
>>
>> http://hadoop.apache.org/common/docs/current/commands_manual.html#jar
>>
>> Thanks
>> Devaraj
>> ________________________________________
>> From: Bejoy Ks [[EMAIL PROTECTED]]
>> Sent: Wednesday, April 04, 2012 1:06 PM
>> To: [EMAIL PROTECTED]
>> Subject: Re: Including third party jar files in Map Reduce job
>>
>> Hi Utkarsh
>> You can add third party jars to your map reduce job elegantly
>> in the following ways
>>
>> 1) use - libjars
>> hadoop jar jarName.jar com.driver.ClassName -libjars /home/some/dir/somejar.jar ....
>>
>> 2) include the third pary jars in /lib folder while packaging your
>> application
>>
>> 3) If you are adding the jar in HADOOP_HOME/lib , you need to add this at all nodes.
>>
>> Regards
>> Bejoy KS
>>
>> On Wed, Apr 4, 2012 at 12:55 PM, Utkarsh Gupta <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>> wrote:
>> Hi Devaraj,
>>
>> I have already copied the required jar file in $HADOOP_HOME/lib folder.
>> Can you tell me where to add generic option -libjars
>>
>> The stack trace is:
>> hadoop$ bin/hadoop jar WordCount.jar /user/hduser1/input/
Harsh J
-
RE: Including third party jar files in Map Reduce jobUtkarsh Gupta 2012-04-04, 13:13
Hi Harsh,
This worked this was exactly what I was looking for.
The warning has gone and now I can add third party jar files using DistributedCache.addFileToClassPath() method.
Now there is no need to copy jar to each node's $HADOOP_HOME/lib folder
Thanks a lot
Utkarsh
-----Original Message-----
From: Harsh J [mailto:[EMAIL PROTECTED]]
Sent: Wednesday, April 04, 2012 6:32 PM
To: [EMAIL PROTECTED]
Subject: Re: Including third party jar files in Map Reduce job
When using Tool, do not use:
Configuration conf = new Configuration();
Instead get config from the class:
Configuration conf = getConf();
This is documented at
http://hadoop.apache.org/common/docs/current/api/org/apache/hadoop/util/Tool.html
On Wed, Apr 4, 2012 at 6:25 PM, Utkarsh Gupta <[EMAIL PROTECTED]> wrote:
> Hi Harsh,
> I have implemented Tool like this
>
> public static void main(String[] args) throws Exception {
> Configuration configuration = new Configuration();
> int rc = ToolRunner.run(configuration, new WordCount(), args);
> System.exit(rc);
> }
>
> @Override
> public int run(String[] args) throws Exception {
> if (args.length < 2) {
> System.err.println("Usage: WordCount <input path> <output
> path>");
> return -1;
> }
> Configuration conf = new Configuration();
> //conf.set("mapred.job.tracker", "local");
> Job job = new Job(conf, "wordcount");
>
> job.setJarByClass(WordCount.class);
> job.setMapperClass(MyMapper.class);
> job.setReducerClass(MyReducer.class);
> job.setOutputKeyClass(Text.class);
> job.setOutputValueClass(IntWritable.class);
> job.setInputFormatClass(TextInputFormat.class);
> job.setOutputFormatClass(TextOutputFormat.class);
> job.setNumReduceTasks(1);
> FileInputFormat.addInputPath(job, new Path(args[0]));
> FileOutputFormat.setOutputPath(job, new Path(args[1]));
> return (job.waitForCompletion(true)) ? 0 : 1;
> }
>
> This is working but I am unable to figure out why still it is warning
> -----Original Message-----
> From: Harsh J [mailto:[EMAIL PROTECTED]]
> Sent: Wednesday, April 04, 2012 6:20 PM
> To: [EMAIL PROTECTED]
> Subject: Re: Including third party jar files in Map Reduce job
>
> Utkarsh,
>
> A log like "12/04/04 15:21:00 WARN mapred.JobClient: Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same." indicates you haven't implemented the Tool approach properly (or aren't calling its run()).
>
> On Wed, Apr 4, 2012 at 5:25 PM, Utkarsh Gupta <[EMAIL PROTECTED]> wrote:
>> Hi Devaraj,
>>
>> The code is running now after copying jar @ each node.
>> I might be doing some mistake previously.
>> Thanks Devaraj and Bejoy :)
>>
>>
>> -----Original Message-----
>> From: Devaraj k [mailto:[EMAIL PROTECTED]]
>> Sent: Wednesday, April 04, 2012 2:08 PM
>> To: [EMAIL PROTECTED]
>> Subject: RE: Including third party jar files in Map Reduce job
>>
>> As Bejoy mentioned,
>>
>> If you have copied the jar to $HADOOP_HOME, then you should copy it
>> to all the nodes in the cluster. (or)
>>
>> If you want to make use of -libjar option, your application should implement Tool to support generic options. Please check the below link for more details.
>>
>> http://hadoop.apache.org/common/docs/current/commands_manual.html#jar
>>
>> Thanks
>> Devaraj
>> ________________________________________
>> From: Bejoy Ks [[EMAIL PROTECTED]]
>> Sent: Wednesday, April 04, 2012 1:06 PM
>> To: [EMAIL PROTECTED]
>> Subject: Re: Including third party jar files in Map Reduce job
>>
>> Hi Utkarsh
>> You can add third party jars to your map reduce job elegantly
>> in the following ways
>>
>> 1) use - libjars
>> hadoop jar jarName.jar com.driver.ClassName -libjars /home/some/dir/somejar.jar ....
>>
>> 2) include the third pary jars in /lib folder while packaging your
Harsh J
This worked this was exactly what I was looking for.
The warning has gone and now I can add third party jar files using DistributedCache.addFileToClassPath() method.
Now there is no need to copy jar to each node's $HADOOP_HOME/lib folder
Thanks a lot
Utkarsh
-----Original Message-----
From: Harsh J [mailto:[EMAIL PROTECTED]]
Sent: Wednesday, April 04, 2012 6:32 PM
To: [EMAIL PROTECTED]
Subject: Re: Including third party jar files in Map Reduce job
When using Tool, do not use:
Configuration conf = new Configuration();
Instead get config from the class:
Configuration conf = getConf();
This is documented at
http://hadoop.apache.org/common/docs/current/api/org/apache/hadoop/util/Tool.html
On Wed, Apr 4, 2012 at 6:25 PM, Utkarsh Gupta <[EMAIL PROTECTED]> wrote:
> Hi Harsh,
> I have implemented Tool like this
>
> public static void main(String[] args) throws Exception {
> Configuration configuration = new Configuration();
> int rc = ToolRunner.run(configuration, new WordCount(), args);
> System.exit(rc);
> }
>
> @Override
> public int run(String[] args) throws Exception {
> if (args.length < 2) {
> System.err.println("Usage: WordCount <input path> <output
> path>");
> return -1;
> }
> Configuration conf = new Configuration();
> //conf.set("mapred.job.tracker", "local");
> Job job = new Job(conf, "wordcount");
>
> job.setJarByClass(WordCount.class);
> job.setMapperClass(MyMapper.class);
> job.setReducerClass(MyReducer.class);
> job.setOutputKeyClass(Text.class);
> job.setOutputValueClass(IntWritable.class);
> job.setInputFormatClass(TextInputFormat.class);
> job.setOutputFormatClass(TextOutputFormat.class);
> job.setNumReduceTasks(1);
> FileInputFormat.addInputPath(job, new Path(args[0]));
> FileOutputFormat.setOutputPath(job, new Path(args[1]));
> return (job.waitForCompletion(true)) ? 0 : 1;
> }
>
> This is working but I am unable to figure out why still it is warning
> -----Original Message-----
> From: Harsh J [mailto:[EMAIL PROTECTED]]
> Sent: Wednesday, April 04, 2012 6:20 PM
> To: [EMAIL PROTECTED]
> Subject: Re: Including third party jar files in Map Reduce job
>
> Utkarsh,
>
> A log like "12/04/04 15:21:00 WARN mapred.JobClient: Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same." indicates you haven't implemented the Tool approach properly (or aren't calling its run()).
>
> On Wed, Apr 4, 2012 at 5:25 PM, Utkarsh Gupta <[EMAIL PROTECTED]> wrote:
>> Hi Devaraj,
>>
>> The code is running now after copying jar @ each node.
>> I might be doing some mistake previously.
>> Thanks Devaraj and Bejoy :)
>>
>>
>> -----Original Message-----
>> From: Devaraj k [mailto:[EMAIL PROTECTED]]
>> Sent: Wednesday, April 04, 2012 2:08 PM
>> To: [EMAIL PROTECTED]
>> Subject: RE: Including third party jar files in Map Reduce job
>>
>> As Bejoy mentioned,
>>
>> If you have copied the jar to $HADOOP_HOME, then you should copy it
>> to all the nodes in the cluster. (or)
>>
>> If you want to make use of -libjar option, your application should implement Tool to support generic options. Please check the below link for more details.
>>
>> http://hadoop.apache.org/common/docs/current/commands_manual.html#jar
>>
>> Thanks
>> Devaraj
>> ________________________________________
>> From: Bejoy Ks [[EMAIL PROTECTED]]
>> Sent: Wednesday, April 04, 2012 1:06 PM
>> To: [EMAIL PROTECTED]
>> Subject: Re: Including third party jar files in Map Reduce job
>>
>> Hi Utkarsh
>> You can add third party jars to your map reduce job elegantly
>> in the following ways
>>
>> 1) use - libjars
>> hadoop jar jarName.jar com.driver.ClassName -libjars /home/some/dir/somejar.jar ....
>>
>> 2) include the third pary jars in /lib folder while packaging your
Harsh J
-
Re: Including third party jar files in Map Reduce jobIoan Eugen Stan 2012-04-04, 15:06
Pe 04.04.2012 16:01, Harsh J a scris:
> When using Tool, do not use:
>
> Configuration conf = new Configuration();
>
> Instead get config from the class:
>
> Configuration conf = getConf();
>
> This is documented at
> http://hadoop.apache.org/common/docs/current/api/org/apache/hadoop/util/Tool.html
>
I wish I knew that when I needed it, but it's not too late know either.
Thanks Harsh, you're doing a great job.
I will update my post, because I had the same problem and a lot of
people run into it.
Regards,
--
Ioan Eugen Stan
http://ieugen.blogspot.com
> When using Tool, do not use:
>
> Configuration conf = new Configuration();
>
> Instead get config from the class:
>
> Configuration conf = getConf();
>
> This is documented at
> http://hadoop.apache.org/common/docs/current/api/org/apache/hadoop/util/Tool.html
>
I wish I knew that when I needed it, but it's not too late know either.
Thanks Harsh, you're doing a great job.
I will update my post, because I had the same problem and a lot of
people run into it.
Regards,
--
Ioan Eugen Stan
http://ieugen.blogspot.com
-
RE: Including third party jar files in Map Reduce jobGUOJUN Zhu 2012-04-04, 17:13
Just a note for "-libjars". All the hadoop Tools options "-libjars",
"-files"... have to be in front of the customized options. Any options
after the one that Tools does not understand will be considered as
customized options and ignored.
Zhu, Guojun
Modeling Sr Graduate
571-3824370
[EMAIL PROTECTED]
Financial Engineering
Freddie Mac
Devaraj k <[EMAIL PROTECTED]>
04/04/2012 04:38 AM
Please respond to
[EMAIL PROTECTED]
To
"[EMAIL PROTECTED]" <[EMAIL PROTECTED]>
cc
Subject
RE: Including third party jar files in Map Reduce job
As Bejoy mentioned,
If you have copied the jar to $HADOOP_HOME, then you should copy it to all
the nodes in the cluster. (or)
If you want to make use of -libjar option, your application should
implement Tool to support generic options. Please check the below link for
more details.
http://hadoop.apache.org/common/docs/current/commands_manual.html#jar
Thanks
Devaraj
________________________________________
From: Bejoy Ks [[EMAIL PROTECTED]]
Sent: Wednesday, April 04, 2012 1:06 PM
To: [EMAIL PROTECTED]
Subject: Re: Including third party jar files in Map Reduce job
Hi Utkarsh
You can add third party jars to your map reduce job elegantly in
the following ways
1) use - libjars
hadoop jar jarName.jar com.driver.ClassName -libjars
/home/some/dir/somejar.jar ....
2) include the third pary jars in /lib folder while packaging your
application
3) If you are adding the jar in HADOOP_HOME/lib , you need to add this at
all nodes.
Regards
Bejoy KS
On Wed, Apr 4, 2012 at 12:55 PM, Utkarsh Gupta <[EMAIL PROTECTED]<
mailto:[EMAIL PROTECTED]>> wrote:
Hi Devaraj,
I have already copied the required jar file in $HADOOP_HOME/lib folder.
Can you tell me where to add generic option -libjars
The stack trace is:
hadoop$ bin/hadoop jar WordCount.jar /user/hduser1/input/
/user/hduser1/output
12/04/04 12:45:51 WARN mapred.JobClient: Use GenericOptionsParser for
parsing the arguments. Applications should implement Tool for the same.
12/04/04 12:45:51 INFO input.FileInputFormat: Total input paths to process
: 1
12/04/04 12:45:51 INFO mapred.JobClient: Running job:
job_201204041107_0005
12/04/04 12:45:52 INFO mapred.JobClient: map 0% reduce 0%
12/04/04 12:46:07 INFO mapred.JobClient: Task Id :
attempt_201204041107_0005_m_000000_0, Status : FAILED
Error: java.lang.ClassNotFoundException:
org.apache.commons.math3.random.RandomDataImpl
at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
at wordcount.MyMapper.map(MyMapper.java:22)
at wordcount.MyMapper.map(MyMapper.java:14)
at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:763)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:369)
at org.apache.hadoop.mapred.Child$4.run(Child.java:259)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1059)
at org.apache.hadoop.mapred.Child.main(Child.java:253)
Thanks and Regards
Utkarsh
-----Original Message-----
From: Devaraj k [mailto:[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>]
Sent: Wednesday, April 04, 2012 12:35 PM
To: [EMAIL PROTECTED]<
mailto:[EMAIL PROTECTED]>
Subject: RE: Including third party jar files in Map Reduce job
Hi Utkarsh,
The usage of the jar command is like this,
Usage: hadoop jar <jar> [mainClass] args...
If you want the commons-math3.jar to be available for all the tasks you
can do any one of these 1. Copy the jar file in $HADOOP_HOME/lib dir or 2.
Use the generic option -libjars.
Can you give the stack trace of your problem for which class it is giving
ClassNotFoundException(i.e for main class or math lib class)?
Thanks
Devaraj
________________________________________
From: Utkarsh Gupta [[EMAIL PROTECTED]<
mailto:[EMAIL PROTECTED]>]
Sent: Wednesday, April 04, 2012 12:22 PM
To: [EMAIL PROTECTED]<
mailto:[EMAIL PROTECTED]>
Subject: Including third party jar files in Map Reduce job
Hi All,
I am new to Hadoop and was trying to generate random numbers using apache
commons math library.
I used Netbeans to build the jar file and the manifest has path to
commons-math jar as lib/commons-math3.jar I have placed this jar file in
HADOOP_HOME/lib folder but still I am getting ClassNotFoundException.
I tried using -libjars option with $HADOOP_HOME/bin/Hadoop jar myprg.jar
<inputpath> <outputpath> -libjars <jarpath> And $HADOOP_HOME/bin/Hadoop
jar myprg.jar -libjar <jarpath> <inputpath> <outputpath> But this is not
working. Please help.
Thanks and Regards
Utkarsh Gupta
**************** CAUTION - Disclaimer ***************** This e-mail
contains PRIVILEGED AND CONFIDENTIAL INFORMATION intended solely for the
use of the addressee(s). If you are not the intended recipient, please
notify the sender by e-mail and delete the original message. Further, you
are not to copy, disclose, or distribute this e-mail or its contents to
any other person and any such actions are unlawful. This e-mail may
contain viruses. Infosys has taken every reasonable precaution to minimize
this risk, but is not liable for any damage you may sustain as a result of
any virus in this e-mail. You should carry out your own virus checks
before opening the e-mail or attachment. Infosys reserves the right to
monitor and review the content of all me
"-files"... have to be in front of the customized options. Any options
after the one that Tools does not understand will be considered as
customized options and ignored.
Zhu, Guojun
Modeling Sr Graduate
571-3824370
[EMAIL PROTECTED]
Financial Engineering
Freddie Mac
Devaraj k <[EMAIL PROTECTED]>
04/04/2012 04:38 AM
Please respond to
[EMAIL PROTECTED]
To
"[EMAIL PROTECTED]" <[EMAIL PROTECTED]>
cc
Subject
RE: Including third party jar files in Map Reduce job
As Bejoy mentioned,
If you have copied the jar to $HADOOP_HOME, then you should copy it to all
the nodes in the cluster. (or)
If you want to make use of -libjar option, your application should
implement Tool to support generic options. Please check the below link for
more details.
http://hadoop.apache.org/common/docs/current/commands_manual.html#jar
Thanks
Devaraj
________________________________________
From: Bejoy Ks [[EMAIL PROTECTED]]
Sent: Wednesday, April 04, 2012 1:06 PM
To: [EMAIL PROTECTED]
Subject: Re: Including third party jar files in Map Reduce job
Hi Utkarsh
You can add third party jars to your map reduce job elegantly in
the following ways
1) use - libjars
hadoop jar jarName.jar com.driver.ClassName -libjars
/home/some/dir/somejar.jar ....
2) include the third pary jars in /lib folder while packaging your
application
3) If you are adding the jar in HADOOP_HOME/lib , you need to add this at
all nodes.
Regards
Bejoy KS
On Wed, Apr 4, 2012 at 12:55 PM, Utkarsh Gupta <[EMAIL PROTECTED]<
mailto:[EMAIL PROTECTED]>> wrote:
Hi Devaraj,
I have already copied the required jar file in $HADOOP_HOME/lib folder.
Can you tell me where to add generic option -libjars
The stack trace is:
hadoop$ bin/hadoop jar WordCount.jar /user/hduser1/input/
/user/hduser1/output
12/04/04 12:45:51 WARN mapred.JobClient: Use GenericOptionsParser for
parsing the arguments. Applications should implement Tool for the same.
12/04/04 12:45:51 INFO input.FileInputFormat: Total input paths to process
: 1
12/04/04 12:45:51 INFO mapred.JobClient: Running job:
job_201204041107_0005
12/04/04 12:45:52 INFO mapred.JobClient: map 0% reduce 0%
12/04/04 12:46:07 INFO mapred.JobClient: Task Id :
attempt_201204041107_0005_m_000000_0, Status : FAILED
Error: java.lang.ClassNotFoundException:
org.apache.commons.math3.random.RandomDataImpl
at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
at wordcount.MyMapper.map(MyMapper.java:22)
at wordcount.MyMapper.map(MyMapper.java:14)
at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:763)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:369)
at org.apache.hadoop.mapred.Child$4.run(Child.java:259)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1059)
at org.apache.hadoop.mapred.Child.main(Child.java:253)
Thanks and Regards
Utkarsh
-----Original Message-----
From: Devaraj k [mailto:[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>]
Sent: Wednesday, April 04, 2012 12:35 PM
To: [EMAIL PROTECTED]<
mailto:[EMAIL PROTECTED]>
Subject: RE: Including third party jar files in Map Reduce job
Hi Utkarsh,
The usage of the jar command is like this,
Usage: hadoop jar <jar> [mainClass] args...
If you want the commons-math3.jar to be available for all the tasks you
can do any one of these 1. Copy the jar file in $HADOOP_HOME/lib dir or 2.
Use the generic option -libjars.
Can you give the stack trace of your problem for which class it is giving
ClassNotFoundException(i.e for main class or math lib class)?
Thanks
Devaraj
________________________________________
From: Utkarsh Gupta [[EMAIL PROTECTED]<
mailto:[EMAIL PROTECTED]>]
Sent: Wednesday, April 04, 2012 12:22 PM
To: [EMAIL PROTECTED]<
mailto:[EMAIL PROTECTED]>
Subject: Including third party jar files in Map Reduce job
Hi All,
I am new to Hadoop and was trying to generate random numbers using apache
commons math library.
I used Netbeans to build the jar file and the manifest has path to
commons-math jar as lib/commons-math3.jar I have placed this jar file in
HADOOP_HOME/lib folder but still I am getting ClassNotFoundException.
I tried using -libjars option with $HADOOP_HOME/bin/Hadoop jar myprg.jar
<inputpath> <outputpath> -libjars <jarpath> And $HADOOP_HOME/bin/Hadoop
jar myprg.jar -libjar <jarpath> <inputpath> <outputpath> But this is not
working. Please help.
Thanks and Regards
Utkarsh Gupta
**************** CAUTION - Disclaimer ***************** This e-mail
contains PRIVILEGED AND CONFIDENTIAL INFORMATION intended solely for the
use of the addressee(s). If you are not the intended recipient, please
notify the sender by e-mail and delete the original message. Further, you
are not to copy, disclose, or distribute this e-mail or its contents to
any other person and any such actions are unlawful. This e-mail may
contain viruses. Infosys has taken every reasonable precaution to minimize
this risk, but is not liable for any damage you may sustain as a result of
any virus in this e-mail. You should carry out your own virus checks
before opening the e-mail or attachment. Infosys reserves the right to
monitor and review the content of all me
