Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Pig >> mail # dev >> Moving to new e2e harness for end-to-end testing

Copy link to this message
RE: Moving to new e2e harness for end-to-end testing
I have a suggestion. Since Pig is also available on Amazon's EMR, can we get some kind folks at Amazon to donate computing resources. It's a win-win - Pig is validated against their infrastructure and the community benefits too.

Any thoughts from the Amazon folks on the mailing list?


-----Original Message-----
From: Alan Gates [mailto:[EMAIL PROTECTED]]
Sent: Thursday, September 01, 2011 10:29 PM
Subject: Re: Moving to new e2e harness for end-to-end testing

One of my goals is to find a way to run public nightly builds for Pig.  Since it needs a cluster it's not clear to me Apache's infrastructure is the best choice.  I'm also investigating OSU's supercell (http://supercell.osuosl.org/) facilities.  Other recommendations are welcome.  I'd also like to get it set up so that there is a patch build, where contributors can submit a patch and a list of tests they think should be run, and have it run for them.

For now I am running the tests nightly on a local Jenkins and will report any errors when I see them.  And there's EC2 for running your own tests if you don't have a cluster.  But this isn't a viable long term solution.


On Sep 1, 2011, at 6:49 PM, Dmitriy Ryaboy wrote:

> Alan,
> Great work.
> Any plans for hooking this up to the apache Jenkins instance?
> D
> On Thu, Sep 1, 2011 at 5:50 PM, Alan Gates <[EMAIL PROTECTED]> wrote:
>> I have gotten the end-to-end test harness to the point where it runs
>> basically all the existing tests and where it can be run from ant.  
>> It can be run on either an existing cluster or in Amazon's EC2.  
>> There are instructions on how to run it in both settings at
>> https://cwiki.apache.org/confluence/display/PIG/HowToTest.  Please
>> try it out to see if it works in your environment.
>> As proposed in
>> https://cwiki.apache.org/confluence/display/PIG/PigTestProposal, I
>> would like to start migrating the junit tests that are really
>> end-to-end tests (ie the ones that use MiniCluster) to the e2e
>> harness.  I'll start with the very long running ones and work down to the shorter ones.
>> Also, as we contribute new patches these should include end-to-end
>> tests that will run in the new harness rather than in junit.  True
>> unit tests should still, of course, be done in junit.
>> Alan.