Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Pig >> mail # dev >> Unit test classpath trouble


Copy link to this message
-
Re: Unit test classpath trouble
It happens consistently on multiple systems and different operating systems
and JDK versions, so while is a build (or more specifically a unit test
environment) problem, it is unlikely an issue singular to me.

On Saturday, May 11, 2013, Konstantin Boudnik wrote:

> As you know we've been testing Pig 0.11 vs 2.0.4-alpha as a part of
> Bigtop's
> validation for the latest hadoop release and it worked ok. Bigtop doesn't
> run
> unit tests though, so it seems like a build issue to me.
>
> Cos
>
> On Sat, May 11, 2013 at 09:19AM, Andrew Purtell wrote:
> > I've tried that, thanks. I did a bit more investigation and it seems the
> > issue is recent Hadoop 2 releases. Has anyone tried running Pig unit
> tests
> > using a more recent Hadoop release than 2.0.0-alpha? Maybe my trouble is
> a
> > simple thing that someone with more experience with Pig internals would
> see
> > right away? Cluster testing seems ok. It's just unit tests that fail. But
> > that is concerning.
> >
> > I'm trying HEAD of branch-0.11.
> >
> > My Java is version "1.6.0_43" Java(TM) SE Runtime Environment (build
> > 1.6.0_43-b01) Java HotSpot(TM) 64-Bit Server VM (build 20.14-b01, mixed
> > mode). OS is Ubuntu 13.04 (GNU/Linux 3.8.0-19-generic x86_64).
> >
> > With defaults and only -Dhadoopversion=23 on the Ant command line, it
> seems
> > ok.
> >
> > With build.properties of:
> >
> > hadoopversion=23
> > hadoop-common.version=2.0.4-alpha
> > hadoop-hdfs.version=2.0.4-alpha
> > hadoop-mapreduce.version=2.0.4-alpha
> >
> >
> > or defined on the Ant command line, I'll see unit test failures like:
> >
> > Testcase: testAccumWithDistinct took 0.868 sec
> >         Caused an ERROR
> > org/apache/hadoop/mapred/ResourceMgrDelegate
> > java.lang.NoClassDefFoundError:
> org/apache/hadoop/mapred/ResourceMgrDelegate
> >         at
> org.apache.hadoop.mapred.YARNRunner.<init>(YARNRunner.java:112)
> >         at
> >
> org.apache.hadoop.mapred.YarnClientProtocolProvider.create(YarnClientProtocolProvider.java:34)
> >         at
> org.apache.hadoop.mapreduce.Cluster.initialize(Cluster.java:94)
> >         at org.apache.hadoop.mapreduce.Cluster.<init>(Cluster.java:81)
> >         at org.apache.hadoop.mapreduce.Cluster.<init>(Cluster.java:74)
> >         at org.apache.hadoop.mapred.JobClient.init(JobClient.java:482)
> >         at org.apache.hadoop.mapred.JobClient.<init>(JobClient.java:461)
> >         at
> >
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher.launchPig(MapReduceLauncher.java:152)
> >         at org.apache.pig.PigServer.launchPlan(PigServer.java:1264)
> >         at
> > org.apache.pig.PigServer.executeCompiledLogicalPlan(PigServer.java:1249)
> >         at org.apache.pig.PigServer.storeEx(PigServer.java:931)
> >         at org.apache.pig.PigServer.store(PigServer.java:898)
> >         at org.apache.pig.PigServer.openIterator(PigServer.java:811)
> >         at
> >
> org.apache.pig.test.TestAccumulator.testAccumWithDistinct(TestAccumulator.java:424)
> >
> > That suggests a cause but I've not started spelunking code with the hope
> > this is something simple that someone has already encountered.
> >
> >
> > On Sat, May 11, 2013 at 1:31 AM, Johnny Zhang <[EMAIL PROTECTED]>
> wrote:
> >
> > > Hi, Andrew:
> > > Does something like "-Dhadoopversion=23" help ? eg. ant clean test
> > > -Dhadoopversion=23 -Dtest.junit.output.format=xml
> > >
> > > Johnny
> > >
> > >
> > > On Fri, May 10, 2013 at 3:39 AM, Andrew Purtell <[EMAIL PROTECTED]>
> > > wrote:
> > >
> > > > Please pardon the basic question. I'm building Pig 0.11.2-SNAPSHOT
> > > against
> > > > Hadoop 2.0.4. 'ant package' and full cluster tests work fine, but
> I'm not
> > > > having much luck with running the unit tests, 'ant test-core' or 'ant
> > > > test'. The problem looks to be a MR app classpath issue.
> > > >
> > > > Sometimes: java.lang.NoClassDefFoundError:
> > > > org/apache/hadoop/yarn/client/YarnClientImpl
> > > >
> > > > Sometimes: java.lang.NoClassDefFoundError:
>
Best regards,

   - Andy

Problems worthy of attack prove their worth by hitting back. - Piet Hein
(via Tom White)
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB