I think there would be many advantages to having a REST API in Kafka, as
a core component. Not just for data services (producing/consuming
messages), but also for admin. I could imagine a Kafka web console that
showed various stats, allowed for maintenance operations, showed pretty
viz of the cluster, etc.
The REST prototype I put together using Jetty was really a simplistic
POC. In reality, more modern web techniques like chunked transfer
encoding and streaming should be used for high throughput (in, and out).
Netty would probably be a good choice for a networking library.
I'd be willing to bet that a sufficiently high performance and easy to
use REST API could put a lot of the 3rd party client libraries out of
To more directly answer your question, Jay, I think if something like
this is done it should be done as a top-level component in Kafka. Is
contrib even maintained/updated anymore? Hasn't Camus supplanted the
On 7/19/13 4:24 PM, Jay Kreps wrote:
> I think this is really cool and would be useful for a lot of people. David,
> do you think it would really benefit from being a contrib package in kafka
> versus a separately hosted github project?
> Here was our experience: contrib packages tend to get added but the authors
> and don't necessarily want involvement in kafka internals or necessarily
> follow the mailing list to answer questions. When we take then they end up
> kind of stagnating because there is overhead for the code review process,
> and the main kafka committers aren't generally qualified to fully maintain
> the code.
> What we have seen is that external projects that host add ons do really
> well. The author tends to get more credit for their work, directly responds
> to questions from users, etc.
> On the other hand maybe we feel this is central enough that it should just
> be a top-level package in kafka. I think this makes sense for things which
> are closely tied to internals, are general enough that there is only one
> good way to do it, and will be very common for kafka users to want. If this
> sounds better for the REST proxy the important thing is just that someone
> has bandwidth to be able to support it and ensure we we document it, have
> sufficient tests to keep it in sync, etc.
> Let me know what you think, if the preference is to add it to kafka then I
> can help with code reviews and stuff.
> On Thu, Jul 18, 2013 at 8:15 PM, David Arthur <[EMAIL PROTECTED]> wrote:
>> I did some work on this a while ago: https://github.com/mumrah/**
>> kafka/commits/rest <https://github.com/mumrah/kafka/commits/rest>
>> I think people would still be interested. I know I would :)
>> On 7/18/13 6:19 PM, Yogita Bijani wrote:
>>> I am a newbie, I was just checking if there has been any updates on the
>>> REST proxy.
>>> I have tried to come up with a draft of REST API and can post it for
>>> review and discussion if it is still an open topic.
>>> Sent from my iPad