Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Pig >> mail # dev >> Unit test classpath trouble


Copy link to this message
-
Re: Unit test classpath trouble
Hi, Andrew:
I just set up a job to run unit test against 2.0.4-alpha. I will
investigate failure and reply to thread.

Thanks,
Johnny Zhang
On Fri, May 10, 2013 at 6:19 PM, Andrew Purtell <[EMAIL PROTECTED]> wrote:

> I've tried that, thanks. I did a bit more investigation and it seems the
> issue is recent Hadoop 2 releases. Has anyone tried running Pig unit tests
> using a more recent Hadoop release than 2.0.0-alpha? Maybe my trouble is a
> simple thing that someone with more experience with Pig internals would see
> right away? Cluster testing seems ok. It's just unit tests that fail. But
> that is concerning.
>
> I'm trying HEAD of branch-0.11.
>
> My Java is version "1.6.0_43" Java(TM) SE Runtime Environment (build
> 1.6.0_43-b01) Java HotSpot(TM) 64-Bit Server VM (build 20.14-b01, mixed
> mode). OS is Ubuntu 13.04 (GNU/Linux 3.8.0-19-generic x86_64).
>
> With defaults and only -Dhadoopversion=23 on the Ant command line, it seems
> ok.
>
> With build.properties of:
>
> hadoopversion=23
> hadoop-common.version=2.0.4-alpha
> hadoop-hdfs.version=2.0.4-alpha
> hadoop-mapreduce.version=2.0.4-alpha
>
>
> or defined on the Ant command line, I'll see unit test failures like:
>
> Testcase: testAccumWithDistinct took 0.868 sec
>         Caused an ERROR
> org/apache/hadoop/mapred/ResourceMgrDelegate
> java.lang.NoClassDefFoundError:
> org/apache/hadoop/mapred/ResourceMgrDelegate
>         at org.apache.hadoop.mapred.YARNRunner.<init>(YARNRunner.java:112)
>         at
>
> org.apache.hadoop.mapred.YarnClientProtocolProvider.create(YarnClientProtocolProvider.java:34)
>         at org.apache.hadoop.mapreduce.Cluster.initialize(Cluster.java:94)
>         at org.apache.hadoop.mapreduce.Cluster.<init>(Cluster.java:81)
>         at org.apache.hadoop.mapreduce.Cluster.<init>(Cluster.java:74)
>         at org.apache.hadoop.mapred.JobClient.init(JobClient.java:482)
>         at org.apache.hadoop.mapred.JobClient.<init>(JobClient.java:461)
>         at
>
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher.launchPig(MapReduceLauncher.java:152)
>         at org.apache.pig.PigServer.launchPlan(PigServer.java:1264)
>         at
> org.apache.pig.PigServer.executeCompiledLogicalPlan(PigServer.java:1249)
>         at org.apache.pig.PigServer.storeEx(PigServer.java:931)
>         at org.apache.pig.PigServer.store(PigServer.java:898)
>         at org.apache.pig.PigServer.openIterator(PigServer.java:811)
>         at
>
> org.apache.pig.test.TestAccumulator.testAccumWithDistinct(TestAccumulator.java:424)
>
> That suggests a cause but I've not started spelunking code with the hope
> this is something simple that someone has already encountered.
>
>
> On Sat, May 11, 2013 at 1:31 AM, Johnny Zhang <[EMAIL PROTECTED]>
> wrote:
>
> > Hi, Andrew:
> > Does something like "-Dhadoopversion=23" help ? eg. ant clean test
> > -Dhadoopversion=23 -Dtest.junit.output.format=xml
> >
> > Johnny
> >
> >
> > On Fri, May 10, 2013 at 3:39 AM, Andrew Purtell <[EMAIL PROTECTED]>
> > wrote:
> >
> > > Please pardon the basic question. I'm building Pig 0.11.2-SNAPSHOT
> > against
> > > Hadoop 2.0.4. 'ant package' and full cluster tests work fine, but I'm
> not
> > > having much luck with running the unit tests, 'ant test-core' or 'ant
> > > test'. The problem looks to be a MR app classpath issue.
> > >
> > > Sometimes: java.lang.NoClassDefFoundError:
> > > org/apache/hadoop/yarn/client/YarnClientImpl
> > >
> > > Sometimes: java.lang.NoClassDefFoundError:
> > > org/apache/hadoop/mapred/ResourceMgrDelegate
> > >
> > > A few Google searches have turned up no useful pointers. Maybe there is
> > > something simple I am missing? How do you set up for running unit tests
> > on
> > > your dev boxes?
> > >
> > > --
> > > Best regards,
> > >
> > >    - Andy
> > >
> > > Problems worthy of attack prove their worth by hitting back. - Piet
> Hein
> > > (via Tom White)
> > >
> >
>
>
>
> --
> Best regards,
>
>    - Andy
>
> Problems worthy of attack prove their worth by hitting back. - Piet Hein