Hi,

Could anyone please guide me as to how to use the -libjars option in HDFS?

I have added the necessary jar file (the hbase jar - to be precise)  to the classpath of the node where I am starting the job.

The following is the format that i am invoking:
bin/hadoop jar <Our Jar> <MainClass> -libjars <Dependent jars (separated by Commas)> <Arguments to our main class>

bin/hadoop jar /Users/hdp/cvk/myjob.jar  mr2.mr2a.MR2ADriver -libjars /Users/hdp/hadoop/lib/hbase-0.20.6.jar inputmr2a  outputmr2a

Despite this,  I find that I get the java.lang.ClassNotFoundException error! :(
java.lang.RuntimeException: java.lang.RuntimeException: java.lang.ClassNotFoundException: org.apache.hadoop.hbase.io.ImmutableBytesWritable
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:841)
at org.apache.hadoop.mapred.JobConf.getMapOutputValueClass(JobConf.java:551)
at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.(MapTask.java:793)
at org.apache.hadoop.mapred.MapTask$NewOutputCollector.(MapTask.java:524)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:613)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305)
at org.apache.hadoop.mapred.Child.main(Child.java:170)
Caused by: java.lang.RuntimeException: java.lang.ClassNotFoundException: org.apache.hadoop.hbase.io.ImmutableBytesWritable
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:809)
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:833)

 The strange thing is that there is another MR job I have  that runs perfectly with the libjars option! Could anybody tell me what I am doing wrong? One more thing - not sure if it is relevant : I am using the new Hadoop MapReduce API.

Thanks in advance!

Regards,
Krishnakumar.

Refer to Alex Kozlov's answer on 12/11/10

On Tue, Jan 11, 2011 at 10:10 AM, C.V.Krishnakumar Iyer
<[EMAIL PROTECTED]>wrote:

> Hi,
>
> Could anyone please guide me as to how to use the -libjars option in HDFS?
>
> I have added the necessary jar file (the hbase jar - to be precise)  to the
> classpath of the node where I am starting the job.
>
> The following is the format that i am invoking:
> bin/hadoop jar <Our Jar> <MainClass> -libjars <Dependent jars (separated by
> Commas)> <Arguments to our main class>
>
> bin/hadoop jar /Users/hdp/cvk/myjob.jar  mr2.mr2a.MR2ADriver -libjars
> /Users/hdp/hadoop/lib/hbase-0.20.6.jar inputmr2a  outputmr2a
>
> Despite this,  I find that I get the java.lang.ClassNotFoundException
> error! :(
> java.lang.RuntimeException: java.lang.RuntimeException:
> java.lang.ClassNotFoundException:
> org.apache.hadoop.hbase.io.ImmutableBytesWritable
>        at
> org.apache.hadoop.conf.Configuration.getClass(Configuration.java:841)
>        at
> org.apache.hadoop.mapred.JobConf.getMapOutputValueClass(JobConf.java:551)
>        at
> org.apache.hadoop.mapred.MapTask$MapOutputBuffer.(MapTask.java:793)
>        at
> org.apache.hadoop.mapred.MapTask$NewOutputCollector.(MapTask.java:524)
>        at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:613)
>        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305)
>        at org.apache.hadoop.mapred.Child.main(Child.java:170)
> Caused by: java.lang.RuntimeException: java.lang.ClassNotFoundException:
> org.apache.hadoop.hbase.io.ImmutableBytesWritable
>        at
> org.apache.hadoop.conf.Configuration.getClass(Configuration.java:809)
>        at
> org.apache.hadoop.conf.Configuration.getClass(Configuration.java:833)
>
>  The strange thing is that there is another MR job I have  that runs
> perfectly with the libjars option! Could anybody tell me what I am doing
> wrong? One more thing - not sure if it is relevant : I am using the new
> Hadoop MapReduce API.
>
> Thanks in advance!
>
> Regards,
> Krishnakumar.

Hi,

I have tried that as well, using -files <jar file> But it still gives the exact same error. Any other thing that I could try?

Thanks,
Krishna.

On Jan 11, 2011, at 10:23 AM, Ted Yu wrote:

> Refer to Alex Kozlov's answer on 12/11/10
>
> On Tue, Jan 11, 2011 at 10:10 AM, C.V.Krishnakumar Iyer
> <[EMAIL PROTECTED]>wrote:
>
>> Hi,
>>
>> Could anyone please guide me as to how to use the -libjars option in HDFS?
>>
>> I have added the necessary jar file (the hbase jar - to be precise)  to the
>> classpath of the node where I am starting the job.
>>
>> The following is the format that i am invoking:
>> bin/hadoop jar <Our Jar> <MainClass> -libjars <Dependent jars (separated by
>> Commas)> <Arguments to our main class>
>>
>> bin/hadoop jar /Users/hdp/cvk/myjob.jar  mr2.mr2a.MR2ADriver -libjars
>> /Users/hdp/hadoop/lib/hbase-0.20.6.jar inputmr2a  outputmr2a
>>
>> Despite this,  I find that I get the java.lang.ClassNotFoundException
>> error! :(
>> java.lang.RuntimeException: java.lang.RuntimeException:
>> java.lang.ClassNotFoundException:
>> org.apache.hadoop.hbase.io.ImmutableBytesWritable
>>       at
>> org.apache.hadoop.conf.Configuration.getClass(Configuration.java:841)
>>       at
>> org.apache.hadoop.mapred.JobConf.getMapOutputValueClass(JobConf.java:551)
>>       at
>> org.apache.hadoop.mapred.MapTask$MapOutputBuffer.(MapTask.java:793)
>>       at
>> org.apache.hadoop.mapred.MapTask$NewOutputCollector.(MapTask.java:524)
>>       at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:613)
>>       at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305)
>>       at org.apache.hadoop.mapred.Child.main(Child.java:170)
>> Caused by: java.lang.RuntimeException: java.lang.ClassNotFoundException:
>> org.apache.hadoop.hbase.io.ImmutableBytesWritable
>>       at
>> org.apache.hadoop.conf.Configuration.getClass(Configuration.java:809)
>>       at
>> org.apache.hadoop.conf.Configuration.getClass(Configuration.java:833)
>>
>> The strange thing is that there is another MR job I have  that runs
>> perfectly with the libjars option! Could anybody tell me what I am doing
>> wrong? One more thing - not sure if it is relevant : I am using the new
>> Hadoop MapReduce API.
>>
>> Thanks in advance!
>>
>> Regards,
>> Krishnakumar.

Have you implemented GenericOptionsParser?  Do you see your jar in the *
mapred.cache.files* or *tmpjars* parameter in your job.xml file (can view
via a JT Web UI)?

--
Alex Kozlov
Solutions Architect
Cloudera, Inc
twitter: alexvk2009
<http://www.cloudera.com/company/press-center/hadoop-world-nyc/>
On Tue, Jan 11, 2011 at 11:49 AM, C.V.Krishnakumar Iyer
<[EMAIL PROTECTED]>wrote:

> Hi,
>
> I have tried that as well, using -files <jar file> But it still gives the
> exact same error. Any other thing that I could try?
>
> Thanks,
> Krishna.
>
> On Jan 11, 2011, at 10:23 AM, Ted Yu wrote:
>
> > Refer to Alex Kozlov's answer on 12/11/10
> >
> > On Tue, Jan 11, 2011 at 10:10 AM, C.V.Krishnakumar Iyer
> > <[EMAIL PROTECTED]>wrote:
> >
> >> Hi,
> >>
> >> Could anyone please guide me as to how to use the -libjars option in
> HDFS?
> >>
> >> I have added the necessary jar file (the hbase jar - to be precise)  to
> the
> >> classpath of the node where I am starting the job.
> >>
> >> The following is the format that i am invoking:
> >> bin/hadoop jar <Our Jar> <MainClass> -libjars <Dependent jars (separated
> by
> >> Commas)> <Arguments to our main class>
> >>
> >> bin/hadoop jar /Users/hdp/cvk/myjob.jar  mr2.mr2a.MR2ADriver -libjars
> >> /Users/hdp/hadoop/lib/hbase-0.20.6.jar inputmr2a  outputmr2a
> >>
> >> Despite this,  I find that I get the java.lang.ClassNotFoundException
> >> error! :(
> >> java.lang.RuntimeException: java.lang.RuntimeException:
> >> java.lang.ClassNotFoundException:
> >> org.apache.hadoop.hbase.io.ImmutableBytesWritable
> >>       at
> >> org.apache.hadoop.conf.Configuration.getClass(Configuration.java:841)
> >>       at
> >>
> org.apache.hadoop.mapred.JobConf.getMapOutputValueClass(JobConf.java:551)
> >>       at
> >> org.apache.hadoop.mapred.MapTask$MapOutputBuffer.(MapTask.java:793)
> >>       at
> >> org.apache.hadoop.mapred.MapTask$NewOutputCollector.(MapTask.java:524)
> >>       at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:613)
> >>       at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305)
> >>       at org.apache.hadoop.mapred.Child.main(Child.java:170)
> >> Caused by: java.lang.RuntimeException: java.lang.ClassNotFoundException:
> >> org.apache.hadoop.hbase.io.ImmutableBytesWritable
> >>       at
> >> org.apache.hadoop.conf.Configuration.getClass(Configuration.java:809)
> >>       at
> >> org.apache.hadoop.conf.Configuration.getClass(Configuration.java:833)
> >>
> >> The strange thing is that there is another MR job I have  that runs
> >> perfectly with the libjars option! Could anybody tell me what I am doing
> >> wrong? One more thing - not sure if it is relevant : I am using the new
> >> Hadoop MapReduce API.
> >>
> >> Thanks in advance!
> >>
> >> Regards,
> >> Krishnakumar.
>
>

There is also a blog that I recently wrote, if it helps
http://www.cloudera.com/blog/2011/01/how-to-include-third-party-libraries-in-your-map-reduce-job

On Tue, Jan 11, 2011 at 12:33 PM, Alex Kozlov <[EMAIL PROTECTED]> wrote:

> Have you implemented GenericOptionsParser?  Do you see your jar in the *
> mapred.cache.files* or *tmpjars* parameter in your job.xml file (can view
> via a JT Web UI)?
>
> --
> Alex Kozlov
> Solutions Architect
> Cloudera, Inc
> twitter: alexvk2009
> <http://www.cloudera.com/company/press-center/hadoop-world-nyc/>
>
>
> On Tue, Jan 11, 2011 at 11:49 AM, C.V.Krishnakumar Iyer <
> [EMAIL PROTECTED]> wrote:
>
>> Hi,
>>
>> I have tried that as well, using -files <jar file> But it still gives the
>> exact same error. Any other thing that I could try?
>>
>> Thanks,
>> Krishna.
>>
>> On Jan 11, 2011, at 10:23 AM, Ted Yu wrote:
>>
>> > Refer to Alex Kozlov's answer on 12/11/10
>> >
>> > On Tue, Jan 11, 2011 at 10:10 AM, C.V.Krishnakumar Iyer
>> > <[EMAIL PROTECTED]>wrote:
>> >
>> >> Hi,
>> >>
>> >> Could anyone please guide me as to how to use the -libjars option in
>> HDFS?
>> >>
>> >> I have added the necessary jar file (the hbase jar - to be precise)  to
>> the
>> >> classpath of the node where I am starting the job.
>> >>
>> >> The following is the format that i am invoking:
>> >> bin/hadoop jar <Our Jar> <MainClass> -libjars <Dependent jars
>> (separated by
>> >> Commas)> <Arguments to our main class>
>> >>
>> >> bin/hadoop jar /Users/hdp/cvk/myjob.jar  mr2.mr2a.MR2ADriver -libjars
>> >> /Users/hdp/hadoop/lib/hbase-0.20.6.jar inputmr2a  outputmr2a
>> >>
>> >> Despite this,  I find that I get the java.lang.ClassNotFoundException
>> >> error! :(
>> >> java.lang.RuntimeException: java.lang.RuntimeException:
>> >> java.lang.ClassNotFoundException:
>> >> org.apache.hadoop.hbase.io.ImmutableBytesWritable
>> >>       at
>> >> org.apache.hadoop.conf.Configuration.getClass(Configuration.java:841)
>> >>       at
>> >>
>> org.apache.hadoop.mapred.JobConf.getMapOutputValueClass(JobConf.java:551)
>> >>       at
>> >> org.apache.hadoop.mapred.MapTask$MapOutputBuffer.(MapTask.java:793)
>> >>       at
>> >> org.apache.hadoop.mapred.MapTask$NewOutputCollector.(MapTask.java:524)
>> >>       at
>> org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:613)
>> >>       at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305)
>> >>       at org.apache.hadoop.mapred.Child.main(Child.java:170)
>> >> Caused by: java.lang.RuntimeException:
>> java.lang.ClassNotFoundException:
>> >> org.apache.hadoop.hbase.io.ImmutableBytesWritable
>> >>       at
>> >> org.apache.hadoop.conf.Configuration.getClass(Configuration.java:809)
>> >>       at
>> >> org.apache.hadoop.conf.Configuration.getClass(Configuration.java:833)
>> >>
>> >> The strange thing is that there is another MR job I have  that runs
>> >> perfectly with the libjars option! Could anybody tell me what I am
>> doing
>> >> wrong? One more thing - not sure if it is relevant : I am using the new
>> >> Hadoop MapReduce API.
>> >>
>> >> Thanks in advance!
>> >>
>> >> Regards,
>> >> Krishnakumar.
>>
>>
>

Hi,

Thanks a lot! I shall try this once and let you know!

Regards,
Krishna.
On Jan 11, 2011, at 12:48 PM, Alex Kozlov wrote:

> There is also a blog that I recently wrote, if it helps
> http://www.cloudera.com/blog/2011/01/how-to-include-third-party-libraries-in-your-map-reduce-job
>
> On Tue, Jan 11, 2011 at 12:33 PM, Alex Kozlov <[EMAIL PROTECTED]> wrote:
>
>> Have you implemented GenericOptionsParser?  Do you see your jar in the *
>> mapred.cache.files* or *tmpjars* parameter in your job.xml file (can view
>> via a JT Web UI)?
>>
>> --
>> Alex Kozlov
>> Solutions Architect
>> Cloudera, Inc
>> twitter: alexvk2009
>> <http://www.cloudera.com/company/press-center/hadoop-world-nyc/>
>>
>>
>> On Tue, Jan 11, 2011 at 11:49 AM, C.V.Krishnakumar Iyer <
>> [EMAIL PROTECTED]> wrote:
>>
>>> Hi,
>>>
>>> I have tried that as well, using -files <jar file> But it still gives the
>>> exact same error. Any other thing that I could try?
>>>
>>> Thanks,
>>> Krishna.
>>>
>>> On Jan 11, 2011, at 10:23 AM, Ted Yu wrote:
>>>
>>>> Refer to Alex Kozlov's answer on 12/11/10
>>>>
>>>> On Tue, Jan 11, 2011 at 10:10 AM, C.V.Krishnakumar Iyer
>>>> <[EMAIL PROTECTED]>wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> Could anyone please guide me as to how to use the -libjars option in
>>> HDFS?
>>>>>
>>>>> I have added the necessary jar file (the hbase jar - to be precise)  to
>>> the
>>>>> classpath of the node where I am starting the job.
>>>>>
>>>>> The following is the format that i am invoking:
>>>>> bin/hadoop jar <Our Jar> <MainClass> -libjars <Dependent jars
>>> (separated by
>>>>> Commas)> <Arguments to our main class>
>>>>>
>>>>> bin/hadoop jar /Users/hdp/cvk/myjob.jar  mr2.mr2a.MR2ADriver -libjars
>>>>> /Users/hdp/hadoop/lib/hbase-0.20.6.jar inputmr2a  outputmr2a
>>>>>
>>>>> Despite this,  I find that I get the java.lang.ClassNotFoundException
>>>>> error! :(
>>>>> java.lang.RuntimeException: java.lang.RuntimeException:
>>>>> java.lang.ClassNotFoundException:
>>>>> org.apache.hadoop.hbase.io.ImmutableBytesWritable
>>>>>      at
>>>>> org.apache.hadoop.conf.Configuration.getClass(Configuration.java:841)
>>>>>      at
>>>>>
>>> org.apache.hadoop.mapred.JobConf.getMapOutputValueClass(JobConf.java:551)
>>>>>      at
>>>>> org.apache.hadoop.mapred.MapTask$MapOutputBuffer.(MapTask.java:793)
>>>>>      at
>>>>> org.apache.hadoop.mapred.MapTask$NewOutputCollector.(MapTask.java:524)
>>>>>      at
>>> org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:613)
>>>>>      at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305)
>>>>>      at org.apache.hadoop.mapred.Child.main(Child.java:170)
>>>>> Caused by: java.lang.RuntimeException:
>>> java.lang.ClassNotFoundException:
>>>>> org.apache.hadoop.hbase.io.ImmutableBytesWritable
>>>>>      at
>>>>> org.apache.hadoop.conf.Configuration.getClass(Configuration.java:809)
>>>>>      at
>>>>> org.apache.hadoop.conf.Configuration.getClass(Configuration.java:833)
>>>>>
>>>>> The strange thing is that there is another MR job I have  that runs
>>>>> perfectly with the libjars option! Could anybody tell me what I am
>>> doing
>>>>> wrong? One more thing - not sure if it is relevant : I am using the new
>>>>> Hadoop MapReduce API.
>>>>>
>>>>> Thanks in advance!
>>>>>
>>>>> Regards,
>>>>> Krishnakumar.
>>>
>>>
>>

Hi,

Thanks a lot Alex! using GenericOptionsParser solved the issue. Previously I had used Tool and had assumed that it would take care of this.

Regards,
Krishna.
On Jan 11, 2011, at 12:48 PM, Alex Kozlov wrote:

> There is also a blog that I recently wrote, if it helps
> http://www.cloudera.com/blog/2011/01/how-to-include-third-party-libraries-in-your-map-reduce-job
>
> On Tue, Jan 11, 2011 at 12:33 PM, Alex Kozlov <[EMAIL PROTECTED]> wrote:
>
>> Have you implemented GenericOptionsParser?  Do you see your jar in the *
>> mapred.cache.files* or *tmpjars* parameter in your job.xml file (can view
>> via a JT Web UI)?
>>
>> --
>> Alex Kozlov
>> Solutions Architect
>> Cloudera, Inc
>> twitter: alexvk2009
>> <http://www.cloudera.com/company/press-center/hadoop-world-nyc/>
>>
>>
>> On Tue, Jan 11, 2011 at 11:49 AM, C.V.Krishnakumar Iyer <
>> [EMAIL PROTECTED]> wrote:
>>
>>> Hi,
>>>
>>> I have tried that as well, using -files <jar file> But it still gives the
>>> exact same error. Any other thing that I could try?
>>>
>>> Thanks,
>>> Krishna.
>>>
>>> On Jan 11, 2011, at 10:23 AM, Ted Yu wrote:
>>>
>>>> Refer to Alex Kozlov's answer on 12/11/10
>>>>
>>>> On Tue, Jan 11, 2011 at 10:10 AM, C.V.Krishnakumar Iyer
>>>> <[EMAIL PROTECTED]>wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> Could anyone please guide me as to how to use the -libjars option in
>>> HDFS?
>>>>>
>>>>> I have added the necessary jar file (the hbase jar - to be precise)  to
>>> the
>>>>> classpath of the node where I am starting the job.
>>>>>
>>>>> The following is the format that i am invoking:
>>>>> bin/hadoop jar <Our Jar> <MainClass> -libjars <Dependent jars
>>> (separated by
>>>>> Commas)> <Arguments to our main class>
>>>>>
>>>>> bin/hadoop jar /Users/hdp/cvk/myjob.jar  mr2.mr2a.MR2ADriver -libjars
>>>>> /Users/hdp/hadoop/lib/hbase-0.20.6.jar inputmr2a  outputmr2a
>>>>>
>>>>> Despite this,  I find that I get the java.lang.ClassNotFoundException
>>>>> error! :(
>>>>> java.lang.RuntimeException: java.lang.RuntimeException:
>>>>> java.lang.ClassNotFoundException:
>>>>> org.apache.hadoop.hbase.io.ImmutableBytesWritable
>>>>>      at
>>>>> org.apache.hadoop.conf.Configuration.getClass(Configuration.java:841)
>>>>>      at
>>>>>
>>> org.apache.hadoop.mapred.JobConf.getMapOutputValueClass(JobConf.java:551)
>>>>>      at
>>>>> org.apache.hadoop.mapred.MapTask$MapOutputBuffer.(MapTask.java:793)
>>>>>      at
>>>>> org.apache.hadoop.mapred.MapTask$NewOutputCollector.(MapTask.java:524)
>>>>>      at
>>> org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:613)
>>>>>      at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305)
>>>>>      at org.apache.hadoop.mapred.Child.main(Child.java:170)
>>>>> Caused by: java.lang.RuntimeException:
>>> java.lang.ClassNotFoundException:
>>>>> org.apache.hadoop.hbase.io.ImmutableBytesWritable
>>>>>      at
>>>>> org.apache.hadoop.conf.Configuration.getClass(Configuration.java:809)
>>>>>      at
>>>>> org.apache.hadoop.conf.Configuration.getClass(Configuration.java:833)
>>>>>
>>>>> The strange thing is that there is another MR job I have  that runs
>>>>> perfectly with the libjars option! Could anybody tell me what I am
>>> doing
>>>>> wrong? One more thing - not sure if it is relevant : I am using the new
>>>>> Hadoop MapReduce API.
>>>>>
>>>>> Thanks in advance!
>>>>>
>>>>> Regards,
>>>>> Krishnakumar.
>>>
>>>
>>