Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Pig, mail # dev - Unit test classpath trouble


Copy link to this message
-
Re: Unit test classpath trouble
Konstantin Boudnik 2013-05-14, 04:15
Cheolsoo,

just to make sure I do not misinterpret your words:
 - there were a change in YARN packaging between 2.0.0 and 2.0.3 that move
   some APIs into separate artifact hadoop-yarn-client
 - adding hadoop-yarn-client is only relevant for 2.0.3 and later and won't
   work on 2.0.0-2.0.2 versions

Does that represent your point of view correctly? If so, increamenting the
version does't look like a big deal anyway, 'cause we are talking about
_minor_ Hadoop version increment, normally reflecting bug fixes, etc.

If the above mistates the situation that I don't see why the version needs to
be bumped up at all.

Please advise. With regards,
  Cos

On Mon, May 13, 2013 at 09:35AM, Cheolsoo Park wrote:
> Hi Cos,
>
> I am happy to backport it to branch-011. But I am not entirely sure whether
> we should bump the Hadoop version in a minor release. As of now,
> branch-0.11 unit tests run fine with 2.0.0-alpha.
>
> Please let me know if anyone has an opinion on this.
>
> Thanks,
> Cheolsoo
>
>
> On Sun, May 12, 2013 at 11:45 PM, Konstantin Boudnik <[EMAIL PROTECTED]> wrote:
>
> > If unit tests can't be run - which seems to be the case - it sound like a
> > release blocker to me. Any plans to back port this fix into 0.11.2?
> > 0.11.1 doesn't have it, right?
> >
> > Cos
> >
> > On Sun, May 12, 2013 at 09:12PM, Cheolsoo Park wrote:
> > > Hi Andrew,
> > >
> > > >> I'm trying HEAD of branch-0.11.
> > >
> > > You need PIG-3248: https://issues.apache.org/jira/browse/PIG-3248. It's
> > not
> > > in branch-0.11 but in trunk.
> > >
> > > In particular, you need to add the following lines to ivy.xml:
> > >
> > > +    <dependency org="org.apache.hadoop" name="hadoop-yarn-client"
> > > +      rev="${hadoop-mapreduce.version}" conf="hadoop23->master"/>
> > >
> > > Thanks,
> > > Cheolsoo
> > >
> > >
> > >
> > > On Sun, May 12, 2013 at 7:13 PM, Andrew Purtell <[EMAIL PROTECTED]>
> > wrote:
> > >
> > > > It happens consistently on multiple systems and different operating
> > systems
> > > > and JDK versions, so while is a build (or more specifically a unit test
> > > > environment) problem, it is unlikely an issue singular to me.
> > > >
> > > > On Saturday, May 11, 2013, Konstantin Boudnik wrote:
> > > >
> > > > > As you know we've been testing Pig 0.11 vs 2.0.4-alpha as a part of
> > > > > Bigtop's
> > > > > validation for the latest hadoop release and it worked ok. Bigtop
> > doesn't
> > > > > run
> > > > > unit tests though, so it seems like a build issue to me.
> > > > >
> > > > > Cos
> > > > >
> > > > > On Sat, May 11, 2013 at 09:19AM, Andrew Purtell wrote:
> > > > > > I've tried that, thanks. I did a bit more investigation and it
> > seems
> > > > the
> > > > > > issue is recent Hadoop 2 releases. Has anyone tried running Pig
> > unit
> > > > > tests
> > > > > > using a more recent Hadoop release than 2.0.0-alpha? Maybe my
> > trouble
> > > > is
> > > > > a
> > > > > > simple thing that someone with more experience with Pig internals
> > would
> > > > > see
> > > > > > right away? Cluster testing seems ok. It's just unit tests that
> > fail.
> > > > But
> > > > > > that is concerning.
> > > > > >
> > > > > > I'm trying HEAD of branch-0.11.
> > > > > >
> > > > > > My Java is version "1.6.0_43" Java(TM) SE Runtime Environment
> > (build
> > > > > > 1.6.0_43-b01) Java HotSpot(TM) 64-Bit Server VM (build 20.14-b01,
> > mixed
> > > > > > mode). OS is Ubuntu 13.04 (GNU/Linux 3.8.0-19-generic x86_64).
> > > > > >
> > > > > > With defaults and only -Dhadoopversion=23 on the Ant command line,
> > it
> > > > > seems
> > > > > > ok.
> > > > > >
> > > > > > With build.properties of:
> > > > > >
> > > > > > hadoopversion=23
> > > > > > hadoop-common.version=2.0.4-alpha
> > > > > > hadoop-hdfs.version=2.0.4-alpha
> > > > > > hadoop-mapreduce.version=2.0.4-alpha
> > > > > >
> > > > > >
> > > > > > or defined on the Ant command line, I'll see unit test failures
> > like:
> > > > > >
> > > > > > Testcase: testAccumWithDistinct took 0.868 sec
> > > > > >         Caused an ERROR