Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka, mail # dev - admin functionality refactoring

Copy link to this message
Re: admin functionality refactoring
Neha Narkhede 2013-02-12, 10:37
The term admin functionality itself means something that only an admin
should be able to do. We had been through a discussion on admin
functionality earlier and reviewed the following options -

1. Expose admin commands as RPC and in addition to that, provide command
line tools that talk to the controller using this RPC. This way non-java
clients also have an option of executing the admin commands from code.
2. Expose admin commands as command line tools only and notify the
controller using zookeeper. The argument is that since these are "admin"
operations, there is no reason someone should be allowed to fire an API
request and change cluster wide metadata, configuration or state. Another
assumption is that since admin operations are limited, this won't create
too much zookeeper state.

The reason we picked option #1 is because it seemed reasonable for admin
operations to be available through command line tools and to disallow
random clients from being able to change cluster state.

However, it's not the end of the world, if we think our users like option
#2 now, we can think about redesigning our admin functionality. It really
depends on what most people think the meaning of admin operations is or
should be.


On Monday, February 11, 2013, S Ahmed wrote:

> I think that is eating too much of your dog food :)
> On Mon, Feb 11, 2013 at 3:08 PM, Jay Kreps <[EMAIL PROTECTED]<javascript:;>>
> wrote:
> > One thought I had on some of these things was that it is actually kind of
> > funny that we implement them in zookeeper. Technically we could implement
> > them in Kafka itself which would mean less code (potentially) and also a
> > clean api to clients.
> >
> > The idea is that many of the admin apis are basically changing cluster
> > state. Examples are adding a topic, deleting a topic, altering the log
> > config for a topic, changing the replication factor for a topic,
> migrating
> > partitions, etc.
> >
> > One way to implement these would be to define an admin topic in kafka.
> Then
> > standardize a JSON format or something for these command type messages.
> The
> > API to make these changes would just be to publish a message to this
> topic.
> > This would allow all brokers to subscribe and carry out these commands in
> > order, and would allow any client that has support for the producer api
> to
> > publish commands.
> >
> > I think in addition to providing an API this is actually the
> functionality
> > needed for most of these cluster changes. For example the dynamic config
> > patch I posted is basically implementing a queue like this but in zk for
> > subscribing to and applying config changes.
> >
> > This is not fully thought through, but thought I would just toss the idea
> > out there.
> >
> > -Jay
> >
> >
> > On Fri, Feb 8, 2013 at 3:12 PM, Jay Kreps <[EMAIL PROTECTED]<javascript:;>>
> wrote:
> >
> > > Admin functionality in 0.8 is kind of messy.  What should we do?
> > >
> > > There are a bunch of misc. command line tools under kafka.admin.
> > > AdminUtils is the closest thing we have to an admin interface, but it
> is
> > > weirdly implemented as all static methods. Furthermore its APIs are
> > really
> > > wonky and tend to describe the operation they perform in zookeeper
> rather
> > > than the high-level thing they accomplish.
> > >
> > > Since people are going to want programmatic access to administrative
> > > functionality it would be good to think this through a bit. The least
> we
> > > could do would be to refactor AdminUtils into an Admin class and think
> > > through those APIs enough to make them usable from java/scala. This
> would
> > > mean having a clear API for each thing that the various tools currently
> > do
> > > (create/delete/alter/describe/list topics, migrate partitions, shutdown
> > > broker, etc).
> > >
> > > Arguably some of these should really be RPC apis so other languages can
> > > invoke basic operations like creating and deleting topics. But this is
> > not