Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Pig >> mail # dev >> Unit test classpath trouble


+
Andrew Purtell 2013-05-10, 10:39
+
Johnny Zhang 2013-05-10, 17:31
+
Andrew Purtell 2013-05-11, 01:19
+
Konstantin Boudnik 2013-05-11, 05:57
+
Andrew Purtell 2013-05-13, 02:13
+
Cheolsoo Park 2013-05-13, 04:12
+
Andrew Purtell 2013-05-13, 08:15
+
Konstantin Boudnik 2013-05-13, 06:45
+
Cheolsoo Park 2013-05-13, 16:35
+
Konstantin Boudnik 2013-05-14, 04:15
+
Johnny Zhang 2013-05-11, 01:49
+
Johnny Zhang 2013-05-13, 17:22
+
Rohini Palaniswamy 2013-05-13, 18:55
Copy link to this message
-
Re: Unit test classpath trouble
Thanks, Rohini, the tests all pass. As Cheolsoo suggested, trunk is fine
with Hadoop 2.0.4-alpha.

Johnny
On Mon, May 13, 2013 at 11:55 AM, Rohini Palaniswamy <
[EMAIL PROTECTED]> wrote:

> Johnny,
>    You need to increase PermGen space for 2.x else TestGrunt will hang with
> the OOM. Run ant with -DdebugArgs="-XX:+CMSClassUnloadingEnabled
> -XX:MaxPermSize=128M"
>
> Regards,
> Rohini
>
>
> On Mon, May 13, 2013 at 10:22 AM, Johnny Zhang <[EMAIL PROTECTED]>
> wrote:
>
> > I run unit test against 2.0.4-alpha in the trunk. The tests hit
> > "OutOfMemoryError:
> > PermGen space" in the middle, but so far no failures or errors. I
> re-kicked
> > off the test.
> >
> > Johnny
> >
> >
> > On Fri, May 10, 2013 at 6:49 PM, Johnny Zhang <[EMAIL PROTECTED]>
> > wrote:
> >
> > > Hi, Andrew:
> > > I just set up a job to run unit test against 2.0.4-alpha. I will
> > > investigate failure and reply to thread.
> > >
> > > Thanks,
> > > Johnny Zhang
> > >
> > >
> > > On Fri, May 10, 2013 at 6:19 PM, Andrew Purtell <[EMAIL PROTECTED]
> > >wrote:
> > >
> > >> I've tried that, thanks. I did a bit more investigation and it seems
> the
> > >> issue is recent Hadoop 2 releases. Has anyone tried running Pig unit
> > tests
> > >> using a more recent Hadoop release than 2.0.0-alpha? Maybe my trouble
> > is a
> > >> simple thing that someone with more experience with Pig internals
> would
> > >> see
> > >> right away? Cluster testing seems ok. It's just unit tests that fail.
> > But
> > >> that is concerning.
> > >>
> > >> I'm trying HEAD of branch-0.11.
> > >>
> > >> My Java is version "1.6.0_43" Java(TM) SE Runtime Environment (build
> > >> 1.6.0_43-b01) Java HotSpot(TM) 64-Bit Server VM (build 20.14-b01,
> mixed
> > >> mode). OS is Ubuntu 13.04 (GNU/Linux 3.8.0-19-generic x86_64).
> > >>
> > >> With defaults and only -Dhadoopversion=23 on the Ant command line, it
> > >> seems
> > >> ok.
> > >>
> > >> With build.properties of:
> > >>
> > >> hadoopversion=23
> > >> hadoop-common.version=2.0.4-alpha
> > >> hadoop-hdfs.version=2.0.4-alpha
> > >> hadoop-mapreduce.version=2.0.4-alpha
> > >>
> > >>
> > >> or defined on the Ant command line, I'll see unit test failures like:
> > >>
> > >> Testcase: testAccumWithDistinct took 0.868 sec
> > >>         Caused an ERROR
> > >> org/apache/hadoop/mapred/ResourceMgrDelegate
> > >> java.lang.NoClassDefFoundError:
> > >> org/apache/hadoop/mapred/ResourceMgrDelegate
> > >>         at
> > org.apache.hadoop.mapred.YARNRunner.<init>(YARNRunner.java:112)
> > >>         at
> > >>
> > >>
> >
> org.apache.hadoop.mapred.YarnClientProtocolProvider.create(YarnClientProtocolProvider.java:34)
> > >>         at
> > org.apache.hadoop.mapreduce.Cluster.initialize(Cluster.java:94)
> > >>         at org.apache.hadoop.mapreduce.Cluster.<init>(Cluster.java:81)
> > >>         at org.apache.hadoop.mapreduce.Cluster.<init>(Cluster.java:74)
> > >>         at org.apache.hadoop.mapred.JobClient.init(JobClient.java:482)
> > >>         at
> org.apache.hadoop.mapred.JobClient.<init>(JobClient.java:461)
> > >>         at
> > >>
> > >>
> >
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher.launchPig(MapReduceLauncher.java:152)
> > >>         at org.apache.pig.PigServer.launchPlan(PigServer.java:1264)
> > >>         at
> > >>
> org.apache.pig.PigServer.executeCompiledLogicalPlan(PigServer.java:1249)
> > >>         at org.apache.pig.PigServer.storeEx(PigServer.java:931)
> > >>         at org.apache.pig.PigServer.store(PigServer.java:898)
> > >>         at org.apache.pig.PigServer.openIterator(PigServer.java:811)
> > >>         at
> > >>
> > >>
> >
> org.apache.pig.test.TestAccumulator.testAccumWithDistinct(TestAccumulator.java:424)
> > >>
> > >> That suggests a cause but I've not started spelunking code with the
> hope
> > >> this is something simple that someone has already encountered.
> > >>
> > >>
> > >> On Sat, May 11, 2013 at 1:31 AM, Johnny Zhang <[EMAIL PROTECTED]>
> > >> wrote:
> > >>
> > >> > Hi, Andrew: