Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka, mail # dev - admin functionality refactoring


Copy link to this message
-
Re: admin functionality refactoring
S Ahmed 2013-02-11, 20:43
I think that is eating too much of your dog food :)
On Mon, Feb 11, 2013 at 3:08 PM, Jay Kreps <[EMAIL PROTECTED]> wrote:

> One thought I had on some of these things was that it is actually kind of
> funny that we implement them in zookeeper. Technically we could implement
> them in Kafka itself which would mean less code (potentially) and also a
> clean api to clients.
>
> The idea is that many of the admin apis are basically changing cluster
> state. Examples are adding a topic, deleting a topic, altering the log
> config for a topic, changing the replication factor for a topic, migrating
> partitions, etc.
>
> One way to implement these would be to define an admin topic in kafka. Then
> standardize a JSON format or something for these command type messages. The
> API to make these changes would just be to publish a message to this topic.
> This would allow all brokers to subscribe and carry out these commands in
> order, and would allow any client that has support for the producer api to
> publish commands.
>
> I think in addition to providing an API this is actually the functionality
> needed for most of these cluster changes. For example the dynamic config
> patch I posted is basically implementing a queue like this but in zk for
> subscribing to and applying config changes.
>
> This is not fully thought through, but thought I would just toss the idea
> out there.
>
> -Jay
>
>
> On Fri, Feb 8, 2013 at 3:12 PM, Jay Kreps <[EMAIL PROTECTED]> wrote:
>
> > Admin functionality in 0.8 is kind of messy.  What should we do?
> >
> > There are a bunch of misc. command line tools under kafka.admin.
> > AdminUtils is the closest thing we have to an admin interface, but it is
> > weirdly implemented as all static methods. Furthermore its APIs are
> really
> > wonky and tend to describe the operation they perform in zookeeper rather
> > than the high-level thing they accomplish.
> >
> > Since people are going to want programmatic access to administrative
> > functionality it would be good to think this through a bit. The least we
> > could do would be to refactor AdminUtils into an Admin class and think
> > through those APIs enough to make them usable from java/scala. This would
> > mean having a clear API for each thing that the various tools currently
> do
> > (create/delete/alter/describe/list topics, migrate partitions, shutdown
> > broker, etc).
> >
> > Arguably some of these should really be RPC apis so other languages can
> > invoke basic operations like creating and deleting topics. But this is
> not
> > critical so we can probably do it later.
> >
> > I think it would be good to also think about how we ended up here. I
> think
> > we are not putting in place basic APIs that provide a layer of
> > simplification for common operations. Instead we tend to just be jabbing
> > directly at Zookeeper from different places using ad hoc methods in
> > ZkUtils. This is really hard to understand or change. In general we seem
> to
> > have trouble thinking through APIs.
> >
> > Thoughts?
> >
> > -Jay
> >
> >
>