Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hadoop >> mail # user >> Adding mahout math jar to hadoop mapreduce execution


Copy link to this message
-
Re: Adding mahout math jar to hadoop mapreduce execution
The -libjars feature is needed if you use the classes in your remote code (map and reduce functions). Is it possible you only use it in your main() method?

-Joey

Sent from my iPhone

On Feb 1, 2012, at 3:38, Daniel Quach <[EMAIL PROTECTED]> wrote:

> Thanks, this seems to make it work. In fact, I did not even need to specify lib jars in the command line…should I be worried that it doesn't work that way?
>
> On Jan 31, 2012, at 4:09 PM, Joey Echeverria wrote:
>
>> You also need to add the jar to the classpath so it's available in
>> your main. You can do soemthing like this:
>>
>> HADOOP_CLASSPATH=/usr/local/mahout/math/target/mahout-math-0.6-SNAPSHOT.jar
>> hadoop jar ...
>>
>> -Joey
>>
>> On Tue, Jan 31, 2012 at 1:38 PM, Daniel Quach <[EMAIL PROTECTED]> wrote:
>>> For Hadoop 0.20.203 (the latest stable), is it sufficient to do this to parse the lib jars from the command line?
>>>
>>> public static main (String args[]) {
>>> Configuration conf = new Configuration();
>>> String[] otherArgs = new GenericOptionsParser(conf, args).getRemainingArgs();
>>>
>>> Job job = new Job(conf, "makevector");
>>> job.setJarByClass(MakeVector.class);
>>> // etc other code for mappers/reducers
>>> }
>>>
>>> I'm thinking I am missing a step here which is why it won't load the mahout jar while running the map reduce.
>>>
>>> On Jan 30, 2012, at 11:10 PM, Daniel Quach wrote:
>>>
>>>> I compiled using javac:
>>>>
>>>> javac -classpath :/usr/local/hadoop/hadoop-core-0.20.203.0.jar:/usr/local/hadoop/lib/commons-cli-1.2.jar:/usr/local/mahout/math/target/mahout-math-0.6-SNAPSHOT.jar -d makevector_classes/ MakeVector.java;
>>>>
>>>> If I don't include the mahout-math jar, it gives me a compile error because of DenseVector.
>>>>
>>>>
>>>> On Jan 30, 2012, at 10:42 PM, Prashant Kommireddi wrote:
>>>>
>>>>> How are you building the mapreduce jar? Try not to include the Mahout dist
>>>>> while building MR jar, and include it only on "-libjars" option.
>>>>>
>>>>> On Mon, Jan 30, 2012 at 10:33 PM, Daniel Quach <[EMAIL PROTECTED]> wrote:
>>>>>
>>>>>> I have been compiling my mapreduce with the jars in the classpath, and I
>>>>>> believe I need to also add the jars as an option to -libjars to hadoop.
>>>>>> However, even when I do this, I still get an error complaining about
>>>>>> missing classes at runtime. (Compilation works fine).
>>>>>>
>>>>>> Here is my command:
>>>>>> hadoop jar makevector.jar org.myorg.MakeVector -libjars
>>>>>> /usr/local/mahout/math/target/mahout-math-0.6-SNAPSHOT.jar input/ output/
>>>>>>
>>>>>> This is the error I receive:
>>>>>> Exception in thread "main" java.lang.NoClassDefFoundError:
>>>>>> org/apache/mahout/math/DenseVector
>>>>>>
>>>>>> I wonder if I am using the GenericOptionsParser incorrectly? I'm not sure
>>>>>> if there is a deeper problem here.
>>>>>>
>>>>
>>>
>>
>>
>>
>> --
>> Joseph Echeverria
>> Cloudera, Inc.
>> 443.305.9434
>
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB