Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hadoop >> mail # user >> Doubt in Hadoop


Copy link to this message
-
Re: Re: Re: Re: Doubt in Hadoop
You need to send a jar to the cluster so it can run your code there. Hadoop
doesn't magically know which jar is the one containing your main class, or
that of your mapper/reducer -- so you need to tell it via that call so it
knows which jar file to upload.

- Aaron

On Sun, Nov 29, 2009 at 7:54 AM, <[EMAIL PROTECTED]> wrote:

> Hi,
>   Actually, I just made the change suggested by Aaron and my code worked.
> But I
> still would like to know why does the setJarbyClass() method have to be
> called
> when the Main class and the Map and Reduce classes are in the same package
> ?
>
> Thank You
>
> Abhishek Agrawal
>
> SUNY- Buffalo
> (716-435-7122)
>
> On Sun 11/29/09 10:39 AM , [EMAIL PROTECTED] sent:
> > Hi,
> > I dont set job.setJarByClass(Map.class). But my main class, the map class
> > andthe reduce class are all in the same package. Does this make any
> difference
> > at ordo I still have to call
> >
> > Thank You
> >
> > Abhishek Agrawal
> >
> > SUNY- Buffalo
> > (716-435-7122)
> >
> > On Fri 11/27/09  1:42 PM , Aaron Kimball aaron@clou
> > dera.com sent:> When you set up the Job object, do you
> > call> job.setJarByClass(Map.class)? That will tell
> > Hadoop which jar file to> ship with the job and to use for classloading
> in
> > your code.>
> > > - Aaron
> > > On Thu, Nov 26, 2009 at 11:56 PM,
> > wrote:> Hi,
> > > Â  I am running the job from command
> > line. The job runs fine in the> local mode
> > > but something happens when I try to run the job
> > in the distributed> mode.
> > > Abhishek Agrawal
> > > SUNY- Buffalo
> > > (716-435-7122)
> > > On Fri 11/27/09 Â 2:31 AM , Jeff
> > Zhang  sent:> > Do you run the map reduce job in command
> > line or IDE?  in map> reduce
> > > > mode, you should put the jar containing the
> > map and reduce class> in
> > > > your classpath
> > > > Jeff Zhang
> > > > On Fri, Nov 27, 2009 at 2:19 PM,
> > Â  wrote:> > Hello Everybody,
> > > > Â  Â  Â
> > Â  Â  Â  Â  Â I have
> > a doubt in Haddop and was wondering> if
> > > > anybody has faced a
> > > > similar problem. I have a package called
> > test. Inside that I have> > class called
> > > > A.java, Map.java, Reduce.java. In A.java I
> > have the main method> > where I am trying
> > > > to initialize the jobConf object. I have
> > written> > jobConf.setMapperClass(Map.class) and
> > similarly for the reduce> class
> > > > as well. The
> > > > code works correctly when I run the code
> > locally via> >
> > jobConf.set("mapred.job.tracker","local") but I get an
> > exception> > when I try to
> > > > run this code on my cluster. The stack
> > trace of the exception is> as
> > > > under. I
> > > > cannot understand the problem. Any help
> > would be appreciated.> > java.lang.RuntimeException:
> > java.lang.RuntimeException:> > java.lang.ClassNotFoundException:
> > test.Map> > Â  Â  Â
> > Â at> >
> > >
> > org.apache.hadoop.conf.Configuration.getClass(Configuration.java:752)> >
> Â  Â  Â
> > Â at> >
> > org.apache.hadoop.mapred.JobConf.getMapperClass(JobConf.java:690)> > Â
> Â  Â
> > Â at> >
> > org.apache.hadoop.mapred.MapRunner.configure(MapRunner.java:34)> > Â  Â
> Â
> > Â at> >
> > >
> > org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:58)>
> > Â  Â  Â
> > Â at> >
> > > >
> > >
> >
> org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:83)
> > > > Â  Â  Â
> > Â at>
> > org.apache.hadoop.mapred.MapTask.run(MapTask.java:338)> > Â  Â  Â
> > Â at
> > org.apache.hadoop.mapred.Child.main(Child.java:158)> > Caused by:
> > java.lang.RuntimeException:> >
> > java.lang.ClassNotFoundException:> > Markowitz.covarMatrixMap
> > > > Â  Â  Â
> > Â at> >
> > >
> > org.apache.hadoop.conf.Configuration.getClass(Configuration.java:720)> >
> Â  Â  Â
> > Â at> >
> > >
> > org.apache.hadoop.conf.Configuration.getClass(Configuration.java:744)> >
> Â  Â  Â
> > Â ... 6 more> > Caused by:
> > java.lang.ClassNotFoundException: test.Map> > Â  Â  Â
> > Â at>
> > java.net.URLClassLoader$1.run(URLClassLoader.java:200)> > Â  Â  Â
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB