Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Pig >> mail # dev >> Unit test classpath trouble


Copy link to this message
-
Re: Unit test classpath trouble
Johnny,
   You need to increase PermGen space for 2.x else TestGrunt will hang with
the OOM. Run ant with -DdebugArgs="-XX:+CMSClassUnloadingEnabled
-XX:MaxPermSize=128M"

Regards,
Rohini
On Mon, May 13, 2013 at 10:22 AM, Johnny Zhang <[EMAIL PROTECTED]> wrote:

> I run unit test against 2.0.4-alpha in the trunk. The tests hit
> "OutOfMemoryError:
> PermGen space" in the middle, but so far no failures or errors. I re-kicked
> off the test.
>
> Johnny
>
>
> On Fri, May 10, 2013 at 6:49 PM, Johnny Zhang <[EMAIL PROTECTED]>
> wrote:
>
> > Hi, Andrew:
> > I just set up a job to run unit test against 2.0.4-alpha. I will
> > investigate failure and reply to thread.
> >
> > Thanks,
> > Johnny Zhang
> >
> >
> > On Fri, May 10, 2013 at 6:19 PM, Andrew Purtell <[EMAIL PROTECTED]
> >wrote:
> >
> >> I've tried that, thanks. I did a bit more investigation and it seems the
> >> issue is recent Hadoop 2 releases. Has anyone tried running Pig unit
> tests
> >> using a more recent Hadoop release than 2.0.0-alpha? Maybe my trouble
> is a
> >> simple thing that someone with more experience with Pig internals would
> >> see
> >> right away? Cluster testing seems ok. It's just unit tests that fail.
> But
> >> that is concerning.
> >>
> >> I'm trying HEAD of branch-0.11.
> >>
> >> My Java is version "1.6.0_43" Java(TM) SE Runtime Environment (build
> >> 1.6.0_43-b01) Java HotSpot(TM) 64-Bit Server VM (build 20.14-b01, mixed
> >> mode). OS is Ubuntu 13.04 (GNU/Linux 3.8.0-19-generic x86_64).
> >>
> >> With defaults and only -Dhadoopversion=23 on the Ant command line, it
> >> seems
> >> ok.
> >>
> >> With build.properties of:
> >>
> >> hadoopversion=23
> >> hadoop-common.version=2.0.4-alpha
> >> hadoop-hdfs.version=2.0.4-alpha
> >> hadoop-mapreduce.version=2.0.4-alpha
> >>
> >>
> >> or defined on the Ant command line, I'll see unit test failures like:
> >>
> >> Testcase: testAccumWithDistinct took 0.868 sec
> >>         Caused an ERROR
> >> org/apache/hadoop/mapred/ResourceMgrDelegate
> >> java.lang.NoClassDefFoundError:
> >> org/apache/hadoop/mapred/ResourceMgrDelegate
> >>         at
> org.apache.hadoop.mapred.YARNRunner.<init>(YARNRunner.java:112)
> >>         at
> >>
> >>
> org.apache.hadoop.mapred.YarnClientProtocolProvider.create(YarnClientProtocolProvider.java:34)
> >>         at
> org.apache.hadoop.mapreduce.Cluster.initialize(Cluster.java:94)
> >>         at org.apache.hadoop.mapreduce.Cluster.<init>(Cluster.java:81)
> >>         at org.apache.hadoop.mapreduce.Cluster.<init>(Cluster.java:74)
> >>         at org.apache.hadoop.mapred.JobClient.init(JobClient.java:482)
> >>         at org.apache.hadoop.mapred.JobClient.<init>(JobClient.java:461)
> >>         at
> >>
> >>
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher.launchPig(MapReduceLauncher.java:152)
> >>         at org.apache.pig.PigServer.launchPlan(PigServer.java:1264)
> >>         at
> >> org.apache.pig.PigServer.executeCompiledLogicalPlan(PigServer.java:1249)
> >>         at org.apache.pig.PigServer.storeEx(PigServer.java:931)
> >>         at org.apache.pig.PigServer.store(PigServer.java:898)
> >>         at org.apache.pig.PigServer.openIterator(PigServer.java:811)
> >>         at
> >>
> >>
> org.apache.pig.test.TestAccumulator.testAccumWithDistinct(TestAccumulator.java:424)
> >>
> >> That suggests a cause but I've not started spelunking code with the hope
> >> this is something simple that someone has already encountered.
> >>
> >>
> >> On Sat, May 11, 2013 at 1:31 AM, Johnny Zhang <[EMAIL PROTECTED]>
> >> wrote:
> >>
> >> > Hi, Andrew:
> >> > Does something like "-Dhadoopversion=23" help ? eg. ant clean test
> >> > -Dhadoopversion=23 -Dtest.junit.output.format=xml
> >> >
> >> > Johnny
> >> >
> >> >
> >> > On Fri, May 10, 2013 at 3:39 AM, Andrew Purtell <[EMAIL PROTECTED]>
> >> > wrote:
> >> >
> >> > > Please pardon the basic question. I'm building Pig 0.11.2-SNAPSHOT
> >> > against
> >> > > Hadoop 2.0.4. 'ant package' and full cluster tests work fine, but
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB