Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Kafka >> mail # dev >> Kafka REST interface


+
David Arthur 2012-08-03, 14:41
+
Jonathan Creasy 2012-08-03, 20:13
+
David Arthur 2012-08-06, 12:39
Copy link to this message
-
Re: Kafka REST interface
That is correct, a Consume REST request would be handed off to a Consumer
class which communicates with ZooKeeper and Kafka. The proxy keeps track of
clients, there are a few routes to manipulate the offsets if you wanted to
force a reset, seek to the end, or re-retrieve some data from a topic.

If others agree, I would be happy to collaborate on integrating a REST
interface to Kakfa directly, more as your have proposed, rather than a
proxy. The proxy was low-hanging fruit within Box because we already had a
framework for cranking out a REST API quite easily and a set of Kafka+ZK
classes. It was simply stitching the two together.

-Jonathan

On Mon, Aug 6, 2012 at 5:39 AM, David Arthur <[EMAIL PROTECTED]> wrote:

> I'd be happy to collaborate on this, though it's been a while since I've
> used PHP.
>
> From what it looks like, what you have is a true proxy that runs outside
> of Kafka and translates some REST routes into Kafka client calls. This
> sounds more in line with what the project page describes. What I have
> proposed is more like a translation layer between some REST routes and
> FetchRequests. In this case the client is responsible for managing offsets.
> Using the consumer groups and ZooKeeper would be another nice way of
> consuming messages (which is probably more like what you have).
>
> Any maintainers have feedback on this?
>
> On Aug 3, 2012, at 4:13 PM, Jonathan Creasy wrote:
>
> > I have an internal one working and was hoping to have it open sourced in
> > the next week. The one at Box is based on the CodeIgniter framework, we
> > have about 45 RESTful interfaces built on this framework so I just put
> > together another one for Kafka.
> >
> >
> > Here are my notes, these were pre-dev so may be a little different than
> > what we ended up with.
> >
> > https://cwiki.apache.org/confluence/display/KAFKA/Restful+API+Proposal
> >
> > I will read yours later this afternoon, we should work together.
> >
> > -Jonathan
> >
> >
> > On Fri, Aug 3, 2012 at 7:41 AM, David Arthur <[EMAIL PROTECTED]> wrote:
> >
> >> I'd like to tackle this project (assuming it hasn't been started yet).
> >>
> >> I wrote up some initial thoughts here: https://gist.github.com/3248179
> >>
> >> TLDR;  use Range header for specifying offsets, simple URIs like
> >> /kafka/topics/[topic]/[partition], use for a simple transport of bytes
> >> and/or represent the messages as some media type (text, json, xml)
> >>
> >> Feedback is most welcome (in the Gist or in this thread).
> >>
> >> Cheers!
> >>
> >> -David
>
>
+
David Arthur 2012-08-10, 16:54
+
Jay Kreps 2012-08-10, 19:50
+
David Arthur 2012-08-11, 03:15
+
Taylor Gautier 2012-08-12, 14:39
+
David Arthur 2012-08-24, 16:37
+
David Arthur 2012-09-10, 13:49
+
David Arthur 2012-11-20, 21:08
+
David Arthur 2012-11-20, 22:06
+
Taylor Gautier 2012-11-21, 15:54
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB