Kafka, mail # dev - Re: admin functionality refactoring - 2013-02-11, 20:09
Solr & Elasticsearch trainings in New York & San Francisco [more info][hide]
 Search Hadoop and all its subprojects:

Switch to Threaded View
Copy link to this message
-
Re: admin functionality refactoring
One thought I had on some of these things was that it is actually kind of
funny that we implement them in zookeeper. Technically we could implement
them in Kafka itself which would mean less code (potentially) and also a
clean api to clients.

The idea is that many of the admin apis are basically changing cluster
state. Examples are adding a topic, deleting a topic, altering the log
config for a topic, changing the replication factor for a topic, migrating
partitions, etc.

One way to implement these would be to define an admin topic in kafka. Then
standardize a JSON format or something for these command type messages. The
API to make these changes would just be to publish a message to this topic.
This would allow all brokers to subscribe and carry out these commands in
order, and would allow any client that has support for the producer api to
publish commands.

I think in addition to providing an API this is actually the functionality
needed for most of these cluster changes. For example the dynamic config
patch I posted is basically implementing a queue like this but in zk for
subscribing to and applying config changes.

This is not fully thought through, but thought I would just toss the idea
out there.

-Jay
On Fri, Feb 8, 2013 at 3:12 PM, Jay Kreps <[EMAIL PROTECTED]> wrote:
 
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB