-Re: Keeping zookeeper happy?
Joel Koshy 2012-08-07, 23:14
Here are some comments from Dave. He'll be adding some more details to a
blog post that we can link from that wiki.
For the most part, yes, it is pretty much the obvious, but here's the
short version of some longer details that I really should get into the
wiki that you pointed at:
- Redundancy in the physical/hardware/network layout: try not to put
them all in the same rack, decent (but don't go nuts) hardware, try to
keep redundant power and network paths, etc
- I/O segregation: if you do a lot of write type traffic you'll almost
definitely want the transaction logs on a different disk group than
app logs and snapshots (the write to the zookeeper service has a
synchronous write to disk, which can be slow).
- Application segregation: Unless you really understand the application
patterns of other apps that you want to install on the same box, it
can be a good idea to run zookeeper in isolation (though this can be a
balancing act with the capabilities of the hardware).
- Use care with virtualization: It can work, depending on your cluster
layout and read/write patterns and SLAs, but the tiny overheads
introduced by the virtualization layer can add up and throw off
zookeeper, as it can be very time sensitive
- Zookeeper configuration and monitoring: It's java, make sure you give
it 'enough' heap space (I usually run them with 3-5G, but that's
mostly due to the data set size we have here). Unfortunately I don't
have a good formula for it. As far as monitoring, both JMZ and the 4
letter commands are very useful, they do overlap in some cases (and in
those cases I prefer the 4 letter commands, they seem more
predictable, or at the very least, they work better with the LI
- Don't overbuild the cluster: large clusters, especially in a write
heavy usage pattern, means a lot of intra cluster communication
(quorums on the writes and subsequent cluster member updates), but
don't underbuild it (and risk swamping the cluster).
Overall, I try to keep the zookeeper system as small as will handle the
load (plus standard growth capacity planning) and as simple as possible.
I try not to do anything fancy with the configuration or application
layout as compared to the official release as well as keep it as self
contained as possible. For these reasons, I tend to skip the OS
packaged versions, since it has a tendency to try to put things in the
OS standard hierarchy, which can be 'messy', for want of a better way to
On Tue, Aug 7, 2012 at 12:00 PM, James A. Robinson <
[EMAIL PROTECTED]> wrote:
> Hi folks,
> The operations wiki page
> states, in part
> Zookeeper is essential for the correct operation of Kafka. There are
> a number of things that must be done to keep zookeeper running
> happily as we have learned the hard way, hopefully Dave and Neha
> will add this since I don't know what we did.
> I was wondering if anyone on the list had comments on this topic?
> Were there things beyond what might be considered obvious, e.g.,
> running at least five nodes on separate machines w/ redundant network
> paths, and so forth?
> - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
> James A. Robinson [EMAIL PROTECTED]
> Stanford University HighWire Press http://highwire.stanford.edu/
> +1 650 7237294 (Work) +1 650 7259335 (Fax)