Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Pig >> mail # dev >> Unit test classpath trouble


Copy link to this message
-
Re: Unit test classpath trouble
Hi, Andrew:
I just set up a job to run unit test against 2.0.4-alpha. I will
investigate failure and reply to thread.

Thanks,
Johnny Zhang
On Fri, May 10, 2013 at 6:19 PM, Andrew Purtell <[EMAIL PROTECTED]> wrote:

> I've tried that, thanks. I did a bit more investigation and it seems the
> issue is recent Hadoop 2 releases. Has anyone tried running Pig unit tests
> using a more recent Hadoop release than 2.0.0-alpha? Maybe my trouble is a
> simple thing that someone with more experience with Pig internals would see
> right away? Cluster testing seems ok. It's just unit tests that fail. But
> that is concerning.
>
> I'm trying HEAD of branch-0.11.
>
> My Java is version "1.6.0_43" Java(TM) SE Runtime Environment (build
> 1.6.0_43-b01) Java HotSpot(TM) 64-Bit Server VM (build 20.14-b01, mixed
> mode). OS is Ubuntu 13.04 (GNU/Linux 3.8.0-19-generic x86_64).
>
> With defaults and only -Dhadoopversion=23 on the Ant command line, it seems
> ok.
>
> With build.properties of:
>
> hadoopversion=23
> hadoop-common.version=2.0.4-alpha
> hadoop-hdfs.version=2.0.4-alpha
> hadoop-mapreduce.version=2.0.4-alpha
>
>
> or defined on the Ant command line, I'll see unit test failures like:
>
> Testcase: testAccumWithDistinct took 0.868 sec
>         Caused an ERROR
> org/apache/hadoop/mapred/ResourceMgrDelegate
> java.lang.NoClassDefFoundError:
> org/apache/hadoop/mapred/ResourceMgrDelegate
>         at org.apache.hadoop.mapred.YARNRunner.<init>(YARNRunner.java:112)
>         at
>
> org.apache.hadoop.mapred.YarnClientProtocolProvider.create(YarnClientProtocolProvider.java:34)
>         at org.apache.hadoop.mapreduce.Cluster.initialize(Cluster.java:94)
>         at org.apache.hadoop.mapreduce.Cluster.<init>(Cluster.java:81)
>         at org.apache.hadoop.mapreduce.Cluster.<init>(Cluster.java:74)
>         at org.apache.hadoop.mapred.JobClient.init(JobClient.java:482)
>         at org.apache.hadoop.mapred.JobClient.<init>(JobClient.java:461)
>         at
>
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher.launchPig(MapReduceLauncher.java:152)
>         at org.apache.pig.PigServer.launchPlan(PigServer.java:1264)
>         at
> org.apache.pig.PigServer.executeCompiledLogicalPlan(PigServer.java:1249)
>         at org.apache.pig.PigServer.storeEx(PigServer.java:931)
>         at org.apache.pig.PigServer.store(PigServer.java:898)
>         at org.apache.pig.PigServer.openIterator(PigServer.java:811)
>         at
>
> org.apache.pig.test.TestAccumulator.testAccumWithDistinct(TestAccumulator.java:424)
>
> That suggests a cause but I've not started spelunking code with the hope
> this is something simple that someone has already encountered.
>
>
> On Sat, May 11, 2013 at 1:31 AM, Johnny Zhang <[EMAIL PROTECTED]>
> wrote:
>
> > Hi, Andrew:
> > Does something like "-Dhadoopversion=23" help ? eg. ant clean test
> > -Dhadoopversion=23 -Dtest.junit.output.format=xml
> >
> > Johnny
> >
> >
> > On Fri, May 10, 2013 at 3:39 AM, Andrew Purtell <[EMAIL PROTECTED]>
> > wrote:
> >
> > > Please pardon the basic question. I'm building Pig 0.11.2-SNAPSHOT
> > against
> > > Hadoop 2.0.4. 'ant package' and full cluster tests work fine, but I'm
> not
> > > having much luck with running the unit tests, 'ant test-core' or 'ant
> > > test'. The problem looks to be a MR app classpath issue.
> > >
> > > Sometimes: java.lang.NoClassDefFoundError:
> > > org/apache/hadoop/yarn/client/YarnClientImpl
> > >
> > > Sometimes: java.lang.NoClassDefFoundError:
> > > org/apache/hadoop/mapred/ResourceMgrDelegate
> > >
> > > A few Google searches have turned up no useful pointers. Maybe there is
> > > something simple I am missing? How do you set up for running unit tests
> > on
> > > your dev boxes?
> > >
> > > --
> > > Best regards,
> > >
> > >    - Andy
> > >
> > > Problems worthy of attack prove their worth by hitting back. - Piet
> Hein
> > > (via Tom White)
> > >
> >
>
>
>
> --
> Best regards,
>
>    - Andy
>
> Problems worthy of attack prove their worth by hitting back. - Piet Hein
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB