Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Pig >> mail # dev >> Unit test classpath trouble


+
Andrew Purtell 2013-05-10, 10:39
+
Johnny Zhang 2013-05-10, 17:31
+
Andrew Purtell 2013-05-11, 01:19
+
Konstantin Boudnik 2013-05-11, 05:57
+
Andrew Purtell 2013-05-13, 02:13
Copy link to this message
-
Re: Unit test classpath trouble
Hi Andrew,

>> I'm trying HEAD of branch-0.11.

You need PIG-3248: https://issues.apache.org/jira/browse/PIG-3248. It's not
in branch-0.11 but in trunk.

In particular, you need to add the following lines to ivy.xml:

+    <dependency org="org.apache.hadoop" name="hadoop-yarn-client"
+      rev="${hadoop-mapreduce.version}" conf="hadoop23->master"/>

Thanks,
Cheolsoo

On Sun, May 12, 2013 at 7:13 PM, Andrew Purtell <[EMAIL PROTECTED]> wrote:

> It happens consistently on multiple systems and different operating systems
> and JDK versions, so while is a build (or more specifically a unit test
> environment) problem, it is unlikely an issue singular to me.
>
> On Saturday, May 11, 2013, Konstantin Boudnik wrote:
>
> > As you know we've been testing Pig 0.11 vs 2.0.4-alpha as a part of
> > Bigtop's
> > validation for the latest hadoop release and it worked ok. Bigtop doesn't
> > run
> > unit tests though, so it seems like a build issue to me.
> >
> > Cos
> >
> > On Sat, May 11, 2013 at 09:19AM, Andrew Purtell wrote:
> > > I've tried that, thanks. I did a bit more investigation and it seems
> the
> > > issue is recent Hadoop 2 releases. Has anyone tried running Pig unit
> > tests
> > > using a more recent Hadoop release than 2.0.0-alpha? Maybe my trouble
> is
> > a
> > > simple thing that someone with more experience with Pig internals would
> > see
> > > right away? Cluster testing seems ok. It's just unit tests that fail.
> But
> > > that is concerning.
> > >
> > > I'm trying HEAD of branch-0.11.
> > >
> > > My Java is version "1.6.0_43" Java(TM) SE Runtime Environment (build
> > > 1.6.0_43-b01) Java HotSpot(TM) 64-Bit Server VM (build 20.14-b01, mixed
> > > mode). OS is Ubuntu 13.04 (GNU/Linux 3.8.0-19-generic x86_64).
> > >
> > > With defaults and only -Dhadoopversion=23 on the Ant command line, it
> > seems
> > > ok.
> > >
> > > With build.properties of:
> > >
> > > hadoopversion=23
> > > hadoop-common.version=2.0.4-alpha
> > > hadoop-hdfs.version=2.0.4-alpha
> > > hadoop-mapreduce.version=2.0.4-alpha
> > >
> > >
> > > or defined on the Ant command line, I'll see unit test failures like:
> > >
> > > Testcase: testAccumWithDistinct took 0.868 sec
> > >         Caused an ERROR
> > > org/apache/hadoop/mapred/ResourceMgrDelegate
> > > java.lang.NoClassDefFoundError:
> > org/apache/hadoop/mapred/ResourceMgrDelegate
> > >         at
> > org.apache.hadoop.mapred.YARNRunner.<init>(YARNRunner.java:112)
> > >         at
> > >
> >
> org.apache.hadoop.mapred.YarnClientProtocolProvider.create(YarnClientProtocolProvider.java:34)
> > >         at
> > org.apache.hadoop.mapreduce.Cluster.initialize(Cluster.java:94)
> > >         at org.apache.hadoop.mapreduce.Cluster.<init>(Cluster.java:81)
> > >         at org.apache.hadoop.mapreduce.Cluster.<init>(Cluster.java:74)
> > >         at org.apache.hadoop.mapred.JobClient.init(JobClient.java:482)
> > >         at
> org.apache.hadoop.mapred.JobClient.<init>(JobClient.java:461)
> > >         at
> > >
> >
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher.launchPig(MapReduceLauncher.java:152)
> > >         at org.apache.pig.PigServer.launchPlan(PigServer.java:1264)
> > >         at
> > >
> org.apache.pig.PigServer.executeCompiledLogicalPlan(PigServer.java:1249)
> > >         at org.apache.pig.PigServer.storeEx(PigServer.java:931)
> > >         at org.apache.pig.PigServer.store(PigServer.java:898)
> > >         at org.apache.pig.PigServer.openIterator(PigServer.java:811)
> > >         at
> > >
> >
> org.apache.pig.test.TestAccumulator.testAccumWithDistinct(TestAccumulator.java:424)
> > >
> > > That suggests a cause but I've not started spelunking code with the
> hope
> > > this is something simple that someone has already encountered.
> > >
> > >
> > > On Sat, May 11, 2013 at 1:31 AM, Johnny Zhang <[EMAIL PROTECTED]>
> > wrote:
> > >
> > > > Hi, Andrew:
> > > > Does something like "-Dhadoopversion=23" help ? eg. ant clean test
> > > > -Dhadoopversion=23 -Dtest.junit.output.format=xml
+
Andrew Purtell 2013-05-13, 08:15
+
Konstantin Boudnik 2013-05-13, 06:45
+
Cheolsoo Park 2013-05-13, 16:35
+
Konstantin Boudnik 2013-05-14, 04:15
+
Johnny Zhang 2013-05-11, 01:49
+
Johnny Zhang 2013-05-13, 17:22
+
Rohini Palaniswamy 2013-05-13, 18:55
+
Johnny Zhang 2013-05-14, 17:12
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB