I have been compiling my mapreduce with the jars in the classpath, and I believe I need to also add the jars as an option to -libjars to hadoop. However, even when I do this, I still get an error complaining about missing classes at runtime. (Compilation works fine).

Here is my command:
hadoop jar makevector.jar org.myorg.MakeVector -libjars /usr/local/mahout/math/target/mahout-math-0.6-SNAPSHOT.jar input/ output/

This is the error I receive:
Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/mahout/math/DenseVector

I wonder if I am using the GenericOptionsParser incorrectly? I'm not sure if there is a deeper problem here.

How are you building the mapreduce jar? Try not to include the Mahout dist
while building MR jar, and include it only on "-libjars" option.

On Mon, Jan 30, 2012 at 10:33 PM, Daniel Quach <[EMAIL PROTECTED]> wrote:

> I have been compiling my mapreduce with the jars in the classpath, and I
> believe I need to also add the jars as an option to -libjars to hadoop.
> However, even when I do this, I still get an error complaining about
> missing classes at runtime. (Compilation works fine).
>
> Here is my command:
> hadoop jar makevector.jar org.myorg.MakeVector -libjars
> /usr/local/mahout/math/target/mahout-math-0.6-SNAPSHOT.jar input/ output/
>
> This is the error I receive:
> Exception in thread "main" java.lang.NoClassDefFoundError:
> org/apache/mahout/math/DenseVector
>
> I wonder if I am using the GenericOptionsParser incorrectly? I'm not sure
> if there is a deeper problem here.
>

I compiled using javac:

javac -classpath :/usr/local/hadoop/hadoop-core-0.20.203.0.jar:/usr/local/hadoop/lib/commons-cli-1.2.jar:/usr/local/mahout/math/target/mahout-math-0.6-SNAPSHOT.jar -d makevector_classes/ MakeVector.java;

If I don't include the mahout-math jar, it gives me a compile error because of DenseVector.
On Jan 30, 2012, at 10:42 PM, Prashant Kommireddi wrote:

> How are you building the mapreduce jar? Try not to include the Mahout dist
> while building MR jar, and include it only on "-libjars" option.
>
> On Mon, Jan 30, 2012 at 10:33 PM, Daniel Quach <[EMAIL PROTECTED]> wrote:
>
>> I have been compiling my mapreduce with the jars in the classpath, and I
>> believe I need to also add the jars as an option to -libjars to hadoop.
>> However, even when I do this, I still get an error complaining about
>> missing classes at runtime. (Compilation works fine).
>>
>> Here is my command:
>> hadoop jar makevector.jar org.myorg.MakeVector -libjars
>> /usr/local/mahout/math/target/mahout-math-0.6-SNAPSHOT.jar input/ output/
>>
>> This is the error I receive:
>> Exception in thread "main" java.lang.NoClassDefFoundError:
>> org/apache/mahout/math/DenseVector
>>
>> I wonder if I am using the GenericOptionsParser incorrectly? I'm not sure
>> if there is a deeper problem here.
>>

For Hadoop 0.20.203 (the latest stable), is it sufficient to do this to parse the lib jars from the command line?

public static main (String args[]) {
Configuration conf = new Configuration();
String[] otherArgs = new GenericOptionsParser(conf, args).getRemainingArgs();

Job job = new Job(conf, "makevector");
job.setJarByClass(MakeVector.class);
// etc other code for mappers/reducers
}

I'm thinking I am missing a step here which is why it won't load the mahout jar while running the map reduce.

On Jan 30, 2012, at 11:10 PM, Daniel Quach wrote:

> I compiled using javac:
>
> javac -classpath :/usr/local/hadoop/hadoop-core-0.20.203.0.jar:/usr/local/hadoop/lib/commons-cli-1.2.jar:/usr/local/mahout/math/target/mahout-math-0.6-SNAPSHOT.jar -d makevector_classes/ MakeVector.java;
>
> If I don't include the mahout-math jar, it gives me a compile error because of DenseVector.
>
>
> On Jan 30, 2012, at 10:42 PM, Prashant Kommireddi wrote:
>
>> How are you building the mapreduce jar? Try not to include the Mahout dist
>> while building MR jar, and include it only on "-libjars" option.
>>
>> On Mon, Jan 30, 2012 at 10:33 PM, Daniel Quach <[EMAIL PROTECTED]> wrote:
>>
>>> I have been compiling my mapreduce with the jars in the classpath, and I
>>> believe I need to also add the jars as an option to -libjars to hadoop.
>>> However, even when I do this, I still get an error complaining about
>>> missing classes at runtime. (Compilation works fine).
>>>
>>> Here is my command:
>>> hadoop jar makevector.jar org.myorg.MakeVector -libjars
>>> /usr/local/mahout/math/target/mahout-math-0.6-SNAPSHOT.jar input/ output/
>>>
>>> This is the error I receive:
>>> Exception in thread "main" java.lang.NoClassDefFoundError:
>>> org/apache/mahout/math/DenseVector
>>>
>>> I wonder if I am using the GenericOptionsParser incorrectly? I'm not sure
>>> if there is a deeper problem here.
>>>
>

You also need to add the jar to the classpath so it's available in
your main. You can do soemthing like this:

HADOOP_CLASSPATH=/usr/local/mahout/math/target/mahout-math-0.6-SNAPSHOT.jar
hadoop jar ...

-Joey

On Tue, Jan 31, 2012 at 1:38 PM, Daniel Quach <[EMAIL PROTECTED]> wrote:
> For Hadoop 0.20.203 (the latest stable), is it sufficient to do this to parse the lib jars from the command line?
>
> public static main (String args[]) {
> Configuration conf = new Configuration();
> String[] otherArgs = new GenericOptionsParser(conf, args).getRemainingArgs();
>
> Job job = new Job(conf, "makevector");
> job.setJarByClass(MakeVector.class);
> // etc other code for mappers/reducers
> }
>
> I'm thinking I am missing a step here which is why it won't load the mahout jar while running the map reduce.
>
> On Jan 30, 2012, at 11:10 PM, Daniel Quach wrote:
>
>> I compiled using javac:
>>
>> javac -classpath :/usr/local/hadoop/hadoop-core-0.20.203.0.jar:/usr/local/hadoop/lib/commons-cli-1.2.jar:/usr/local/mahout/math/target/mahout-math-0.6-SNAPSHOT.jar -d makevector_classes/ MakeVector.java;
>>
>> If I don't include the mahout-math jar, it gives me a compile error because of DenseVector.
>>
>>
>> On Jan 30, 2012, at 10:42 PM, Prashant Kommireddi wrote:
>>
>>> How are you building the mapreduce jar? Try not to include the Mahout dist
>>> while building MR jar, and include it only on "-libjars" option.
>>>
>>> On Mon, Jan 30, 2012 at 10:33 PM, Daniel Quach <[EMAIL PROTECTED]> wrote:
>>>
>>>> I have been compiling my mapreduce with the jars in the classpath, and I
>>>> believe I need to also add the jars as an option to -libjars to hadoop.
>>>> However, even when I do this, I still get an error complaining about
>>>> missing classes at runtime. (Compilation works fine).
>>>>
>>>> Here is my command:
>>>> hadoop jar makevector.jar org.myorg.MakeVector -libjars
>>>> /usr/local/mahout/math/target/mahout-math-0.6-SNAPSHOT.jar input/ output/
>>>>
>>>> This is the error I receive:
>>>> Exception in thread "main" java.lang.NoClassDefFoundError:
>>>> org/apache/mahout/math/DenseVector
>>>>
>>>> I wonder if I am using the GenericOptionsParser incorrectly? I'm not sure
>>>> if there is a deeper problem here.
>>>>
>>
>

--
Joseph Echeverria
Cloudera, Inc.
443.305.9434

Thanks, this seems to make it work. In fact, I did not even need to specify lib jars in the command line…should I be worried that it doesn't work that way?

On Jan 31, 2012, at 4:09 PM, Joey Echeverria wrote:

> You also need to add the jar to the classpath so it's available in
> your main. You can do soemthing like this:
>
> HADOOP_CLASSPATH=/usr/local/mahout/math/target/mahout-math-0.6-SNAPSHOT.jar
> hadoop jar ...
>
> -Joey
>
> On Tue, Jan 31, 2012 at 1:38 PM, Daniel Quach <[EMAIL PROTECTED]> wrote:
>> For Hadoop 0.20.203 (the latest stable), is it sufficient to do this to parse the lib jars from the command line?
>>
>> public static main (String args[]) {
>> Configuration conf = new Configuration();
>> String[] otherArgs = new GenericOptionsParser(conf, args).getRemainingArgs();
>>
>> Job job = new Job(conf, "makevector");
>> job.setJarByClass(MakeVector.class);
>> // etc other code for mappers/reducers
>> }
>>
>> I'm thinking I am missing a step here which is why it won't load the mahout jar while running the map reduce.
>>
>> On Jan 30, 2012, at 11:10 PM, Daniel Quach wrote:
>>
>>> I compiled using javac:
>>>
>>> javac -classpath :/usr/local/hadoop/hadoop-core-0.20.203.0.jar:/usr/local/hadoop/lib/commons-cli-1.2.jar:/usr/local/mahout/math/target/mahout-math-0.6-SNAPSHOT.jar -d makevector_classes/ MakeVector.java;
>>>
>>> If I don't include the mahout-math jar, it gives me a compile error because of DenseVector.
>>>
>>>
>>> On Jan 30, 2012, at 10:42 PM, Prashant Kommireddi wrote:
>>>
>>>> How are you building the mapreduce jar? Try not to include the Mahout dist
>>>> while building MR jar, and include it only on "-libjars" option.
>>>>
>>>> On Mon, Jan 30, 2012 at 10:33 PM, Daniel Quach <[EMAIL PROTECTED]> wrote:
>>>>
>>>>> I have been compiling my mapreduce with the jars in the classpath, and I
>>>>> believe I need to also add the jars as an option to -libjars to hadoop.
>>>>> However, even when I do this, I still get an error complaining about
>>>>> missing classes at runtime. (Compilation works fine).
>>>>>
>>>>> Here is my command:
>>>>> hadoop jar makevector.jar org.myorg.MakeVector -libjars
>>>>> /usr/local/mahout/math/target/mahout-math-0.6-SNAPSHOT.jar input/ output/
>>>>>
>>>>> This is the error I receive:
>>>>> Exception in thread "main" java.lang.NoClassDefFoundError:
>>>>> org/apache/mahout/math/DenseVector
>>>>>
>>>>> I wonder if I am using the GenericOptionsParser incorrectly? I'm not sure
>>>>> if there is a deeper problem here.
>>>>>
>>>
>>
>
>
>
> --
> Joseph Echeverria
> Cloudera, Inc.
> 443.305.9434

The -libjars feature is needed if you use the classes in your remote code (map and reduce functions). Is it possible you only use it in your main() method?

-Joey

Sent from my iPhone

On Feb 1, 2012, at 3:38, Daniel Quach <[EMAIL PROTECTED]> wrote:

> Thanks, this seems to make it work. In fact, I did not even need to specify lib jars in the command line…should I be worried that it doesn't work that way?
>
> On Jan 31, 2012, at 4:09 PM, Joey Echeverria wrote:
>
>> You also need to add the jar to the classpath so it's available in
>> your main. You can do soemthing like this:
>>
>> HADOOP_CLASSPATH=/usr/local/mahout/math/target/mahout-math-0.6-SNAPSHOT.jar
>> hadoop jar ...
>>
>> -Joey
>>
>> On Tue, Jan 31, 2012 at 1:38 PM, Daniel Quach <[EMAIL PROTECTED]> wrote:
>>> For Hadoop 0.20.203 (the latest stable), is it sufficient to do this to parse the lib jars from the command line?
>>>
>>> public static main (String args[]) {
>>> Configuration conf = new Configuration();
>>> String[] otherArgs = new GenericOptionsParser(conf, args).getRemainingArgs();
>>>
>>> Job job = new Job(conf, "makevector");
>>> job.setJarByClass(MakeVector.class);
>>> // etc other code for mappers/reducers
>>> }
>>>
>>> I'm thinking I am missing a step here which is why it won't load the mahout jar while running the map reduce.
>>>
>>> On Jan 30, 2012, at 11:10 PM, Daniel Quach wrote:
>>>
>>>> I compiled using javac:
>>>>
>>>> javac -classpath :/usr/local/hadoop/hadoop-core-0.20.203.0.jar:/usr/local/hadoop/lib/commons-cli-1.2.jar:/usr/local/mahout/math/target/mahout-math-0.6-SNAPSHOT.jar -d makevector_classes/ MakeVector.java;
>>>>
>>>> If I don't include the mahout-math jar, it gives me a compile error because of DenseVector.
>>>>
>>>>
>>>> On Jan 30, 2012, at 10:42 PM, Prashant Kommireddi wrote:
>>>>
>>>>> How are you building the mapreduce jar? Try not to include the Mahout dist
>>>>> while building MR jar, and include it only on "-libjars" option.
>>>>>
>>>>> On Mon, Jan 30, 2012 at 10:33 PM, Daniel Quach <[EMAIL PROTECTED]> wrote:
>>>>>
>>>>>> I have been compiling my mapreduce with the jars in the classpath, and I
>>>>>> believe I need to also add the jars as an option to -libjars to hadoop.
>>>>>> However, even when I do this, I still get an error complaining about
>>>>>> missing classes at runtime. (Compilation works fine).
>>>>>>
>>>>>> Here is my command:
>>>>>> hadoop jar makevector.jar org.myorg.MakeVector -libjars
>>>>>> /usr/local/mahout/math/target/mahout-math-0.6-SNAPSHOT.jar input/ output/
>>>>>>
>>>>>> This is the error I receive:
>>>>>> Exception in thread "main" java.lang.NoClassDefFoundError:
>>>>>> org/apache/mahout/math/DenseVector
>>>>>>
>>>>>> I wonder if I am using the GenericOptionsParser incorrectly? I'm not sure
>>>>>> if there is a deeper problem here.
>>>>>>
>>>>
>>>
>>
>>
>>
>> --
>> Joseph Echeverria
>> Cloudera, Inc.
>> 443.305.9434
>