Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # dev >> dynamic config zookeeper code


Copy link to this message
-
Re: dynamic config zookeeper code
TopicConfigWatcher sounds good

I still feel there needs to be some abstraction to TopicConfigWather
because using Zookeeper Watcher means having global changes to a cluster
and in production we don't allow global updates to every server so i am
going to have to use another mechanism.  How we do many configuration
changes in production is to notify a server that ZK has a change and that
one server goes and gets the changes so we can do sequential configuration
changes with some sanity check for each server change is alright. one way
we do this is responding to http requests with name=value (i.e.
http://serverXYZ/admin/config=reload then hit a health check on serverXYZ).
 for scala servers we embed jetty.

if you can abstract the watcher that would be awesome or I can abstract it
after you are done and build another update.  I am not sold my way is
better for anyone that the ZK watcher I actually like that approach but if
others need or want the feature (like we would but i don't mind rolling a
patch into production that is not in the main code base) then we can agree
to support both methods.

also what about having a new package

kafka/config

this new class goes there

and another ticket refactoring other configs into there?  am I getting too
off my rocker to suggest re-packing kafka to org.apache.kafka? I don't mind
the code change in the kafka code but the downstream clients may freek...
but if we want to-do it now would be the time to-do it me thinks

On Thu, Jan 24, 2013 at 12:40 AM, Jay Kreps <[EMAIL PROTECTED]> wrote:

> Where should the zk code for the dynamic topic config go? I am not sure I
> understand the overall lay of the land well enough to know if I am doing
> the right thing.
>
> For instance, what the is KafkaZookeeper? What is its domain of
> responsibility? Should all zk code go there? Lots of other zk code seems
> elsewhere, KafkaZookeeper just seems to be the znode for broker liveness
> (perhaps it should be renamed?). Why does some zk stuff go in
> KafkaZookeeper and other stuff elsewhere? Actually in practice lots of it
> seems to be stuck in ZkUtils as static methods, is there a reason for this?
>
> Maybe this stuff needs a maintainer? Or does it fall under "replication"?
>
> What I am thinking of doing is making a new class TopicConfigWatcher which
> has a readAllLogConfigs() method and a watcher on the config change path.
> This class would update a map of configs which would be shared with
> LogManager to handle "alter topic". "create topic" can just do the read
> from zk in response to the controller. Does this sound reasonable?
>
> -Jay
>

--

/*
Joe Stein
http://www.linkedin.com/in/charmalloc
Twitter: @allthingshadoop <http://www.twitter.com/allthingshadoop>
*/

 
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB