Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Pig >> mail # dev >> Moving to new e2e harness for end-to-end testing


Copy link to this message
-
RE: Moving to new e2e harness for end-to-end testing
I have a suggestion. Since Pig is also available on Amazon's EMR, can we get some kind folks at Amazon to donate computing resources. It's a win-win - Pig is validated against their infrastructure and the community benefits too.

Any thoughts from the Amazon folks on the mailing list?

Thanks,
Santhosh

-----Original Message-----
From: Alan Gates [mailto:[EMAIL PROTECTED]]
Sent: Thursday, September 01, 2011 10:29 PM
To: [EMAIL PROTECTED]
Subject: Re: Moving to new e2e harness for end-to-end testing

One of my goals is to find a way to run public nightly builds for Pig.  Since it needs a cluster it's not clear to me Apache's infrastructure is the best choice.  I'm also investigating OSU's supercell (http://supercell.osuosl.org/) facilities.  Other recommendations are welcome.  I'd also like to get it set up so that there is a patch build, where contributors can submit a patch and a list of tests they think should be run, and have it run for them.

For now I am running the tests nightly on a local Jenkins and will report any errors when I see them.  And there's EC2 for running your own tests if you don't have a cluster.  But this isn't a viable long term solution.

Alan.

On Sep 1, 2011, at 6:49 PM, Dmitriy Ryaboy wrote:

> Alan,
> Great work.
> Any plans for hooking this up to the apache Jenkins instance?
>
> D
>
> On Thu, Sep 1, 2011 at 5:50 PM, Alan Gates <[EMAIL PROTECTED]> wrote:
>
>> I have gotten the end-to-end test harness to the point where it runs
>> basically all the existing tests and where it can be run from ant.  
>> It can be run on either an existing cluster or in Amazon's EC2.  
>> There are instructions on how to run it in both settings at
>> https://cwiki.apache.org/confluence/display/PIG/HowToTest.  Please
>> try it out to see if it works in your environment.
>>
>> As proposed in
>> https://cwiki.apache.org/confluence/display/PIG/PigTestProposal, I
>> would like to start migrating the junit tests that are really
>> end-to-end tests (ie the ones that use MiniCluster) to the e2e
>> harness.  I'll start with the very long running ones and work down to the shorter ones.
>>
>> Also, as we contribute new patches these should include end-to-end
>> tests that will run in the new harness rather than in junit.  True
>> unit tests should still, of course, be done in junit.
>>
>> Alan.
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB