Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Pig, mail # dev - Unit test classpath trouble


Copy link to this message
-
Re: Unit test classpath trouble
Andrew Purtell 2013-05-13, 08:15
Cheolsoo,

Thank you so much. That was the problem.
On Mon, May 13, 2013 at 12:12 PM, Cheolsoo Park <[EMAIL PROTECTED]>wrote:

> Hi Andrew,
>
> >> I'm trying HEAD of branch-0.11.
>
> You need PIG-3248: https://issues.apache.org/jira/browse/PIG-3248. It's
> not
> in branch-0.11 but in trunk.
>
> In particular, you need to add the following lines to ivy.xml:
>
> +    <dependency org="org.apache.hadoop" name="hadoop-yarn-client"
> +      rev="${hadoop-mapreduce.version}" conf="hadoop23->master"/>
>
> Thanks,
> Cheolsoo
>
>
>
> On Sun, May 12, 2013 at 7:13 PM, Andrew Purtell <[EMAIL PROTECTED]>
> wrote:
>
> > It happens consistently on multiple systems and different operating
> systems
> > and JDK versions, so while is a build (or more specifically a unit test
> > environment) problem, it is unlikely an issue singular to me.
> >
> > On Saturday, May 11, 2013, Konstantin Boudnik wrote:
> >
> > > As you know we've been testing Pig 0.11 vs 2.0.4-alpha as a part of
> > > Bigtop's
> > > validation for the latest hadoop release and it worked ok. Bigtop
> doesn't
> > > run
> > > unit tests though, so it seems like a build issue to me.
> > >
> > > Cos
> > >
> > > On Sat, May 11, 2013 at 09:19AM, Andrew Purtell wrote:
> > > > I've tried that, thanks. I did a bit more investigation and it seems
> > the
> > > > issue is recent Hadoop 2 releases. Has anyone tried running Pig unit
> > > tests
> > > > using a more recent Hadoop release than 2.0.0-alpha? Maybe my trouble
> > is
> > > a
> > > > simple thing that someone with more experience with Pig internals
> would
> > > see
> > > > right away? Cluster testing seems ok. It's just unit tests that fail.
> > But
> > > > that is concerning.
> > > >
> > > > I'm trying HEAD of branch-0.11.
> > > >
> > > > My Java is version "1.6.0_43" Java(TM) SE Runtime Environment (build
> > > > 1.6.0_43-b01) Java HotSpot(TM) 64-Bit Server VM (build 20.14-b01,
> mixed
> > > > mode). OS is Ubuntu 13.04 (GNU/Linux 3.8.0-19-generic x86_64).
> > > >
> > > > With defaults and only -Dhadoopversion=23 on the Ant command line, it
> > > seems
> > > > ok.
> > > >
> > > > With build.properties of:
> > > >
> > > > hadoopversion=23
> > > > hadoop-common.version=2.0.4-alpha
> > > > hadoop-hdfs.version=2.0.4-alpha
> > > > hadoop-mapreduce.version=2.0.4-alpha
> > > >
> > > >
> > > > or defined on the Ant command line, I'll see unit test failures like:
> > > >
> > > > Testcase: testAccumWithDistinct took 0.868 sec
> > > >         Caused an ERROR
> > > > org/apache/hadoop/mapred/ResourceMgrDelegate
> > > > java.lang.NoClassDefFoundError:
> > > org/apache/hadoop/mapred/ResourceMgrDelegate
> > > >         at
> > > org.apache.hadoop.mapred.YARNRunner.<init>(YARNRunner.java:112)
> > > >         at
> > > >
> > >
> >
> org.apache.hadoop.mapred.YarnClientProtocolProvider.create(YarnClientProtocolProvider.java:34)
> > > >         at
> > > org.apache.hadoop.mapreduce.Cluster.initialize(Cluster.java:94)
> > > >         at
> org.apache.hadoop.mapreduce.Cluster.<init>(Cluster.java:81)
> > > >         at
> org.apache.hadoop.mapreduce.Cluster.<init>(Cluster.java:74)
> > > >         at
> org.apache.hadoop.mapred.JobClient.init(JobClient.java:482)
> > > >         at
> > org.apache.hadoop.mapred.JobClient.<init>(JobClient.java:461)
> > > >         at
> > > >
> > >
> >
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher.launchPig(MapReduceLauncher.java:152)
> > > >         at org.apache.pig.PigServer.launchPlan(PigServer.java:1264)
> > > >         at
> > > >
> > org.apache.pig.PigServer.executeCompiledLogicalPlan(PigServer.java:1249)
> > > >         at org.apache.pig.PigServer.storeEx(PigServer.java:931)
> > > >         at org.apache.pig.PigServer.store(PigServer.java:898)
> > > >         at org.apache.pig.PigServer.openIterator(PigServer.java:811)
> > > >         at
> > > >
> > >
> >
> org.apache.pig.test.TestAccumulator.testAccumWithDistinct(TestAccumulator.java:424)
> > > >
> > > > That suggests a cause but I've not started spelunking code with the

Best regards,

   - Andy

Problems worthy of attack prove their worth by hitting back. - Piet Hein
(via Tom White)