Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
HBase >> mail # dev >> Re: HBASE-4746


Hi Ted,

I actually just run every test class (Test*.java) in its own mapper using "mvn -Dtest=Test… test" from a script running over Hadoop streaming, and upload test results into HDFS.

The effort to categorize tests into small, medium, and large is very useful. We probably won't do this in 89-fb for now, though. I guess HBASE-4746 would be good enough for parallelizing 89-fb tests at the moment.

Thanks,
--Mikhail

From: Ted Yu <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>>
Date: Sat, 5 Nov 2011 19:08:58 -0700
To: Mikhail Bautin <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>>
Subject: Re: HBASE-4746

Thanks for your message.
Appreciate it.

Take your time - as you indicated HBASE-4746 is not in the critical path for test parallelization.

How does the map-reduce test runner partition the tests ? I assume you also categorize the tests as being small, medium and large (integration) tests.

Cheers

On Sat, Nov 5, 2011 at 6:55 PM, Mikhail Bautin <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>> wrote:
Hi Ted,

The previous total execution time for 89-fb test suite was around 1.5 hours. I used a 100-node cluster to run tests in 10 minutes, but I will try to use a 5-node dev cluster as well and will post the results. By the way, the trunk does not seem to have problems that 89-fb has regarding not passing the correct ZK client port to clients, but I will still prepare a patch similar to the one I have out for 89-fb on HBASE-4746.

Thanks,
--Mikhail

From: Ted Yu <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>>
Date: Fri, 4 Nov 2011 11:41:41 -0700
To: Mikhail Bautin <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>>
Subject: HBASE-4746

Mikhail:
The performance numbers from HBASE-4746 are amazing.
Can you describe the previous total execution time for 0.89 test suite ?
Also, to achieve 10 minute execution time, how many servers were used ?

Thanks

+
Stack 2011-11-06, 21:09
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB