Currently kafka broker config is all statically defined in a properties file with the broker. This mostly works pretty well, but for per-topic configuration (the flush policy, partition count, etc) it is pretty painful to have to bounce the broker every time you make a config change.
An open question is how topic-default configurations should work.
Currently each of our topic-level configs is paired with a default. So you would have something like segment.size.bytes which would be the default, and then you can override this for topics that need something different using a map: segment.size.bytes.per.topic
The proposal is to move the topic configuration into zookeeper so that for a topic "my-topic" we would have a znode /brokers/topics/my-topic/config and the contents of this znode would be the topic configuration either as json or properties or whatever.
There are two ways this config could work: 1. Defaults resolved at topic creation time: At the time a topic is created the user would specify some properties they wanted for that topic, any topic they didn't specify would take the server default. ALL these properties would be stored in the znode. 2. Defaults resolved at config read time: When a topic is created the user specifies particularly properties they want and ONLY the properties they particularly specify would be stored. At runtime we would merge these properties with whatever the server defaults currently are.
This is a somewhat nuanced point, but perhaps important.
The advantage of the first proposal is that it is simple. If you want to know the configuration for a particular topic you go to zookeeper and look at that topics config. Mixing the combination of server config and zookeeper config dynamically makes it a little harder to figure out what the current state of anything is.
The disadvantage of the first proposal (and the advantage of the second proposal) is that making global changes is easier. For example if you want to globally lower the retention for all topics, in proposal one you would have to iterate over all topics and update the config (this could be done automatically with tooling, but under the covers the tool would do this). In the second case you would just update the default value.
Thoughts? If no one cares, I will just pick whatever seems best.
Also can we abstract the config call too? We have so much in chef, it's not that i don't want to call our zookeeper cluster for it but we don't have our topology yet mapped out in znodes they are in our own instances of code.
It should have both a pull and push for changes, one thing that's nice with zookeeper and having a watcher.
I am not sure if I understand what you are proposing, though. Are you saying support both the config file and zk for topic-level configs? I hate to do things where the answer is "do both"...I guess I feel that although everyone walks away happy it ends up being a lot of code and combinatorial testing. So if there is a different plan that hits all requirements I like that better. I am very sensitive to the fact that zookeeper is an okay key/value store but a really poor replacement for a config management system. It might be worth while to try to work out a way that meets all needs, if such a thing exists.
Is bouncing brokers for topic-overrides a problem for you in your environment? If so how would you fix it?
On Fri, Jan 18, 2013 at 7:53 AM, Joe Stein <[EMAIL PROTECTED]> wrote:
Well, but the proposal was that topic-level configs are loaded when you run the create_topic command, so wouldn't that be what you are asking for?
-Jay On Fri, Jan 18, 2013 at 1:57 PM, Joe Stein <[EMAIL PROTECTED]> wrote:
NEW: Monitor These Apps!
Apache Lucene, Apache Solr and all other Apache Software Foundation projects and their respective logos are trademarks of the Apache Software Foundation.
Elasticsearch, Kibana, Logstash, and Beats are trademarks of Elasticsearch BV, registered in the U.S. and in other countries. This site and Sematext Group is in no way affiliated with Elasticsearch BV.
Service operated by Sematext