Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka, mail # user - Re: Unit test failure


Copy link to this message
-
Re: Unit test failure
Chris Burroughs 2011-12-03, 16:32
On 11/30/2011 01:02 AM, Jay Kreps wrote:
> I think this is one of the core problems of this type of system--how to do
> good, realistic distributed testing. I have not seen anyone really succeed
> except via the "throw production load at it and see" approach so I would be
> interested to hear people's thoughts.

Cassandra has the same problem and has gone through several iterations
of distributed test systems.  The most recent is here:
https://github.com/riptano/cassandra-dtest  Time will tell if this
attempt is more successful.  The obvious prerequisite for something like
this is a set of scripts that make it easy to set up Kafka clusters of
different topologies for testing.

Another thing they do well is having a solid stress/perf tool
distributed with releases, everyone agrees it gives decent results, easy
to hack if you want to test something slightly different and thus it's
widely used.