I have a suggestion. Since Pig is also available on Amazon's EMR, can we get some kind folks at Amazon to donate computing resources. It's a win-win - Pig is validated against their infrastructure and the community benefits too.
Any thoughts from the Amazon folks on the mailing list?
From: Alan Gates [mailto:[EMAIL PROTECTED]]
Sent: Thursday, September 01, 2011 10:29 PM
To: [EMAIL PROTECTED]
Subject: Re: Moving to new e2e harness for end-to-end testing
One of my goals is to find a way to run public nightly builds for Pig. Since it needs a cluster it's not clear to me Apache's infrastructure is the best choice. I'm also investigating OSU's supercell (http://supercell.osuosl.org/) facilities. Other recommendations are welcome. I'd also like to get it set up so that there is a patch build, where contributors can submit a patch and a list of tests they think should be run, and have it run for them.
For now I am running the tests nightly on a local Jenkins and will report any errors when I see them. And there's EC2 for running your own tests if you don't have a cluster. But this isn't a viable long term solution.
On Sep 1, 2011, at 6:49 PM, Dmitriy Ryaboy wrote:
> Great work.
> Any plans for hooking this up to the apache Jenkins instance?
> On Thu, Sep 1, 2011 at 5:50 PM, Alan Gates <[EMAIL PROTECTED]> wrote:
>> I have gotten the end-to-end test harness to the point where it runs
>> basically all the existing tests and where it can be run from ant.
>> It can be run on either an existing cluster or in Amazon's EC2.
>> There are instructions on how to run it in both settings at
>> https://cwiki.apache.org/confluence/display/PIG/HowToTest. Please
>> try it out to see if it works in your environment.
>> As proposed in
>> https://cwiki.apache.org/confluence/display/PIG/PigTestProposal, I
>> would like to start migrating the junit tests that are really
>> end-to-end tests (ie the ones that use MiniCluster) to the e2e
>> harness. I'll start with the very long running ones and work down to the shorter ones.
>> Also, as we contribute new patches these should include end-to-end
>> tests that will run in the new harness rather than in junit. True
>> unit tests should still, of course, be done in junit.