Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Pig, mail # dev - Unit test classpath trouble


Copy link to this message
-
Re: Unit test classpath trouble
Johnny Zhang 2013-05-13, 17:22
I run unit test against 2.0.4-alpha in the trunk. The tests hit
"OutOfMemoryError:
PermGen space" in the middle, but so far no failures or errors. I re-kicked
off the test.

Johnny
On Fri, May 10, 2013 at 6:49 PM, Johnny Zhang <[EMAIL PROTECTED]> wrote:

> Hi, Andrew:
> I just set up a job to run unit test against 2.0.4-alpha. I will
> investigate failure and reply to thread.
>
> Thanks,
> Johnny Zhang
>
>
> On Fri, May 10, 2013 at 6:19 PM, Andrew Purtell <[EMAIL PROTECTED]>wrote:
>
>> I've tried that, thanks. I did a bit more investigation and it seems the
>> issue is recent Hadoop 2 releases. Has anyone tried running Pig unit tests
>> using a more recent Hadoop release than 2.0.0-alpha? Maybe my trouble is a
>> simple thing that someone with more experience with Pig internals would
>> see
>> right away? Cluster testing seems ok. It's just unit tests that fail. But
>> that is concerning.
>>
>> I'm trying HEAD of branch-0.11.
>>
>> My Java is version "1.6.0_43" Java(TM) SE Runtime Environment (build
>> 1.6.0_43-b01) Java HotSpot(TM) 64-Bit Server VM (build 20.14-b01, mixed
>> mode). OS is Ubuntu 13.04 (GNU/Linux 3.8.0-19-generic x86_64).
>>
>> With defaults and only -Dhadoopversion=23 on the Ant command line, it
>> seems
>> ok.
>>
>> With build.properties of:
>>
>> hadoopversion=23
>> hadoop-common.version=2.0.4-alpha
>> hadoop-hdfs.version=2.0.4-alpha
>> hadoop-mapreduce.version=2.0.4-alpha
>>
>>
>> or defined on the Ant command line, I'll see unit test failures like:
>>
>> Testcase: testAccumWithDistinct took 0.868 sec
>>         Caused an ERROR
>> org/apache/hadoop/mapred/ResourceMgrDelegate
>> java.lang.NoClassDefFoundError:
>> org/apache/hadoop/mapred/ResourceMgrDelegate
>>         at org.apache.hadoop.mapred.YARNRunner.<init>(YARNRunner.java:112)
>>         at
>>
>> org.apache.hadoop.mapred.YarnClientProtocolProvider.create(YarnClientProtocolProvider.java:34)
>>         at org.apache.hadoop.mapreduce.Cluster.initialize(Cluster.java:94)
>>         at org.apache.hadoop.mapreduce.Cluster.<init>(Cluster.java:81)
>>         at org.apache.hadoop.mapreduce.Cluster.<init>(Cluster.java:74)
>>         at org.apache.hadoop.mapred.JobClient.init(JobClient.java:482)
>>         at org.apache.hadoop.mapred.JobClient.<init>(JobClient.java:461)
>>         at
>>
>> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher.launchPig(MapReduceLauncher.java:152)
>>         at org.apache.pig.PigServer.launchPlan(PigServer.java:1264)
>>         at
>> org.apache.pig.PigServer.executeCompiledLogicalPlan(PigServer.java:1249)
>>         at org.apache.pig.PigServer.storeEx(PigServer.java:931)
>>         at org.apache.pig.PigServer.store(PigServer.java:898)
>>         at org.apache.pig.PigServer.openIterator(PigServer.java:811)
>>         at
>>
>> org.apache.pig.test.TestAccumulator.testAccumWithDistinct(TestAccumulator.java:424)
>>
>> That suggests a cause but I've not started spelunking code with the hope
>> this is something simple that someone has already encountered.
>>
>>
>> On Sat, May 11, 2013 at 1:31 AM, Johnny Zhang <[EMAIL PROTECTED]>
>> wrote:
>>
>> > Hi, Andrew:
>> > Does something like "-Dhadoopversion=23" help ? eg. ant clean test
>> > -Dhadoopversion=23 -Dtest.junit.output.format=xml
>> >
>> > Johnny
>> >
>> >
>> > On Fri, May 10, 2013 at 3:39 AM, Andrew Purtell <[EMAIL PROTECTED]>
>> > wrote:
>> >
>> > > Please pardon the basic question. I'm building Pig 0.11.2-SNAPSHOT
>> > against
>> > > Hadoop 2.0.4. 'ant package' and full cluster tests work fine, but I'm
>> not
>> > > having much luck with running the unit tests, 'ant test-core' or 'ant
>> > > test'. The problem looks to be a MR app classpath issue.
>> > >
>> > > Sometimes: java.lang.NoClassDefFoundError:
>> > > org/apache/hadoop/yarn/client/YarnClientImpl
>> > >
>> > > Sometimes: java.lang.NoClassDefFoundError:
>> > > org/apache/hadoop/mapred/ResourceMgrDelegate
>> > >
>> > > A few Google searches have turned up no useful pointers. Maybe there
>