I'm getting ready to try out this configuration (use multiple disks, no RAID, per broker). One concern is the procedure for recovering if there is a disk failure.
If a disk fails, will the broker go offline, or will it continue serving partitions on its remaining good disks? And if so, is there a procedure for moving the partitions that were on the failed disk, but not necessarily all the others on that broker?
Jason On Thu, Jun 20, 2013 at 3:15 PM, Jason Rosenberg <[EMAIL PROTECTED]> wrote:
If you get a disk error that results in an IOException the broker will shut itself down. You would then have the option of replacing the disk or deleting that data directory from the list. When the broker is brought back up the intact partitions will quickly catch up and be online; the destroyed partitions will have to fully rebuild off the other replicas and will take a little longer but will automatically come back online once they have restored off the replicas.
Sent from my iPhone
On Aug 14, 2013, at 1:49 PM, Jason Rosenberg <[EMAIL PROTECTED]> wrote:
Ok, that makes sense that the broker will shut itself down.
If we bring it back up, can this be with an altered set of log.dirs? Will the destroyed partitions get rebuilt on a new log.dir? Or do we have to bring it back up with a new or repaired disk, matching the old log.dir, in order for those replicas to be rebuilt?
Jason On Wed, Aug 14, 2013 at 4:16 PM, Jay Kreps <[EMAIL PROTECTED]> wrote:
My assumption is that you have something like log.dirs=/mnt/diska,/mnt/diskb,/mnt/diskc
Let's say that diskc goes bad. The expected behavior is that the kafka broker gets an error on write and kills itself.
When that happens you have two options.
Either you remove the drive and add a new one and remount it as /mnt/diskc (but with no data, just an empty directory). When the broker restarts it will pull down the data for the partitions that belong on diskc.
Your other option, which you ask about, is to remove /mnt/diskc from the log.dirs list. If you do this when the broker comes online it will recreate the partitions it needs but will add them to /mnt/diska and /mnt/diskc (roughly round-robin).
Hopefully that makes sense.
-Jay On Thu, Aug 15, 2013 at 11:27 AM, Vadim Keylis <[EMAIL PROTECTED]>wrote:
NEW: Monitor These Apps!
Apache Lucene, Apache Solr and all other Apache Software Foundation projects and their respective logos are trademarks of the Apache Software Foundation.
Elasticsearch, Kibana, Logstash, and Beats are trademarks of Elasticsearch BV, registered in the U.S. and in other countries. This site and Sematext Group is in no way affiliated with Elasticsearch BV.
Service operated by Sematext