I mostly agree with the solution proposed by Asmund. One addition would be
that you can have replicas for each partition and that would give you fault
tolerance. You can use FULL AUTO mode and let Helix manage everything.

Regarding the statement "My understanding of helix is that it isn't trivial
to dynamically add a partition", it is possible to add partitions
dynamically and we heavily rely on that feature in Pinot.

thanks,
Kishore G

On Thu, Jul 12, 2018 at 11:06 AM, Åsmund Tokheim <[EMAIL PROTECTED]> wrote:
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB