Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Pig >> mail # dev >> Unit test classpath trouble


+
Andrew Purtell 2013-05-10, 10:39
+
Johnny Zhang 2013-05-10, 17:31
Copy link to this message
-
Re: Unit test classpath trouble
I've tried that, thanks. I did a bit more investigation and it seems the
issue is recent Hadoop 2 releases. Has anyone tried running Pig unit tests
using a more recent Hadoop release than 2.0.0-alpha? Maybe my trouble is a
simple thing that someone with more experience with Pig internals would see
right away? Cluster testing seems ok. It's just unit tests that fail. But
that is concerning.

I'm trying HEAD of branch-0.11.

My Java is version "1.6.0_43" Java(TM) SE Runtime Environment (build
1.6.0_43-b01) Java HotSpot(TM) 64-Bit Server VM (build 20.14-b01, mixed
mode). OS is Ubuntu 13.04 (GNU/Linux 3.8.0-19-generic x86_64).

With defaults and only -Dhadoopversion=23 on the Ant command line, it seems
ok.

With build.properties of:

hadoopversion=23
hadoop-common.version=2.0.4-alpha
hadoop-hdfs.version=2.0.4-alpha
hadoop-mapreduce.version=2.0.4-alpha
or defined on the Ant command line, I'll see unit test failures like:

Testcase: testAccumWithDistinct took 0.868 sec
        Caused an ERROR
org/apache/hadoop/mapred/ResourceMgrDelegate
java.lang.NoClassDefFoundError: org/apache/hadoop/mapred/ResourceMgrDelegate
        at org.apache.hadoop.mapred.YARNRunner.<init>(YARNRunner.java:112)
        at
org.apache.hadoop.mapred.YarnClientProtocolProvider.create(YarnClientProtocolProvider.java:34)
        at org.apache.hadoop.mapreduce.Cluster.initialize(Cluster.java:94)
        at org.apache.hadoop.mapreduce.Cluster.<init>(Cluster.java:81)
        at org.apache.hadoop.mapreduce.Cluster.<init>(Cluster.java:74)
        at org.apache.hadoop.mapred.JobClient.init(JobClient.java:482)
        at org.apache.hadoop.mapred.JobClient.<init>(JobClient.java:461)
        at
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher.launchPig(MapReduceLauncher.java:152)
        at org.apache.pig.PigServer.launchPlan(PigServer.java:1264)
        at
org.apache.pig.PigServer.executeCompiledLogicalPlan(PigServer.java:1249)
        at org.apache.pig.PigServer.storeEx(PigServer.java:931)
        at org.apache.pig.PigServer.store(PigServer.java:898)
        at org.apache.pig.PigServer.openIterator(PigServer.java:811)
        at
org.apache.pig.test.TestAccumulator.testAccumWithDistinct(TestAccumulator.java:424)

That suggests a cause but I've not started spelunking code with the hope
this is something simple that someone has already encountered.
On Sat, May 11, 2013 at 1:31 AM, Johnny Zhang <[EMAIL PROTECTED]> wrote:

> Hi, Andrew:
> Does something like "-Dhadoopversion=23" help ? eg. ant clean test
> -Dhadoopversion=23 -Dtest.junit.output.format=xml
>
> Johnny
>
>
> On Fri, May 10, 2013 at 3:39 AM, Andrew Purtell <[EMAIL PROTECTED]>
> wrote:
>
> > Please pardon the basic question. I'm building Pig 0.11.2-SNAPSHOT
> against
> > Hadoop 2.0.4. 'ant package' and full cluster tests work fine, but I'm not
> > having much luck with running the unit tests, 'ant test-core' or 'ant
> > test'. The problem looks to be a MR app classpath issue.
> >
> > Sometimes: java.lang.NoClassDefFoundError:
> > org/apache/hadoop/yarn/client/YarnClientImpl
> >
> > Sometimes: java.lang.NoClassDefFoundError:
> > org/apache/hadoop/mapred/ResourceMgrDelegate
> >
> > A few Google searches have turned up no useful pointers. Maybe there is
> > something simple I am missing? How do you set up for running unit tests
> on
> > your dev boxes?
> >
> > --
> > Best regards,
> >
> >    - Andy
> >
> > Problems worthy of attack prove their worth by hitting back. - Piet Hein
> > (via Tom White)
> >
>

--
Best regards,

   - Andy

Problems worthy of attack prove their worth by hitting back. - Piet Hein
(via Tom White)
+
Konstantin Boudnik 2013-05-11, 05:57
+
Andrew Purtell 2013-05-13, 02:13
+
Cheolsoo Park 2013-05-13, 04:12
+
Andrew Purtell 2013-05-13, 08:15
+
Konstantin Boudnik 2013-05-13, 06:45
+
Cheolsoo Park 2013-05-13, 16:35
+
Konstantin Boudnik 2013-05-14, 04:15
+
Johnny Zhang 2013-05-11, 01:49
+
Johnny Zhang 2013-05-13, 17:22
+
Rohini Palaniswamy 2013-05-13, 18:55
+
Johnny Zhang 2013-05-14, 17:12