Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # dev >> REST API


I think there would be many advantages to having a REST API in Kafka, as
a core component. Not just for data services (producing/consuming
messages), but also for admin. I could imagine a Kafka web console that
showed various stats, allowed for maintenance operations, showed pretty
viz of the cluster, etc.

The REST prototype I put together using Jetty was really a simplistic
POC. In reality, more modern web techniques like chunked transfer
encoding and streaming should be used for high throughput (in, and out).
Netty would probably be a good choice for a networking library.

I'd be willing to bet that a sufficiently high performance and easy to
use REST API could put a lot of the 3rd party client libraries out of
business, so-to-speak.

To more directly answer your question, Jay, I think if something like
this is done it should be done as a top-level component in Kafka. Is
contrib even maintained/updated anymore? Hasn't Camus supplanted the
hadoop stuff?

On 7/19/13 4:24 PM, Jay Kreps wrote:
> I think this is really cool and would be useful for a lot of people. David,
> do you think it would really benefit from being a contrib package in kafka
> versus a separately hosted github project?
>
> Here was our experience: contrib packages tend to get added but the authors
> and don't necessarily want involvement in kafka internals or necessarily
> follow the mailing list to answer questions. When we take then they end up
> kind of stagnating because there is overhead for the code review process,
> and the main kafka committers aren't generally qualified to fully maintain
> the code.
>
> What we have seen is that external projects that host add ons do really
> well. The author tends to get more credit for their work, directly responds
> to questions from users, etc.
>
> On the other hand maybe we feel this is central enough that it should just
> be a top-level package in kafka. I think this makes sense for things which
> are closely tied to internals, are general enough that there is only one
> good way to do it, and will be very common for kafka users to want. If this
> sounds better for the REST proxy the important thing is just that someone
> has bandwidth to be able to support it and ensure we we document it, have
> sufficient tests to keep it in sync, etc.
>
> Let me know what you think, if the preference is to add it to kafka then I
> can help with code reviews and stuff.
>
> -Jay
>
>
> On Thu, Jul 18, 2013 at 8:15 PM, David Arthur <[EMAIL PROTECTED]> wrote:
>
>> I did some work on this a while ago: https://github.com/mumrah/**
>> kafka/commits/rest <https://github.com/mumrah/kafka/commits/rest>
>>
>> I think people would still be interested. I know I would :)
>>
>> -David
>>
>>
>>
>> On 7/18/13 6:19 PM, Yogita Bijani wrote:
>>
>>> Hi,
>>> I am a newbie, I was just checking if there has been any updates on the
>>> REST proxy.
>>> I have tried to come up with a draft of REST API and can post it for
>>> review and discussion if it is still an open topic.
>>>
>>> Thanks
>>> Yogita
>>>
>>> Sent from my iPad
>>>
>>
 
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB