Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka, mail # dev - Kafka REST interface


Copy link to this message
-
Re: Kafka REST interface
Jonathan Creasy 2012-08-06, 18:19
That is correct, a Consume REST request would be handed off to a Consumer
class which communicates with ZooKeeper and Kafka. The proxy keeps track of
clients, there are a few routes to manipulate the offsets if you wanted to
force a reset, seek to the end, or re-retrieve some data from a topic.

If others agree, I would be happy to collaborate on integrating a REST
interface to Kakfa directly, more as your have proposed, rather than a
proxy. The proxy was low-hanging fruit within Box because we already had a
framework for cranking out a REST API quite easily and a set of Kafka+ZK
classes. It was simply stitching the two together.

-Jonathan

On Mon, Aug 6, 2012 at 5:39 AM, David Arthur <[EMAIL PROTECTED]> wrote:

> I'd be happy to collaborate on this, though it's been a while since I've
> used PHP.
>
> From what it looks like, what you have is a true proxy that runs outside
> of Kafka and translates some REST routes into Kafka client calls. This
> sounds more in line with what the project page describes. What I have
> proposed is more like a translation layer between some REST routes and
> FetchRequests. In this case the client is responsible for managing offsets.
> Using the consumer groups and ZooKeeper would be another nice way of
> consuming messages (which is probably more like what you have).
>
> Any maintainers have feedback on this?
>
> On Aug 3, 2012, at 4:13 PM, Jonathan Creasy wrote:
>
> > I have an internal one working and was hoping to have it open sourced in
> > the next week. The one at Box is based on the CodeIgniter framework, we
> > have about 45 RESTful interfaces built on this framework so I just put
> > together another one for Kafka.
> >
> >
> > Here are my notes, these were pre-dev so may be a little different than
> > what we ended up with.
> >
> > https://cwiki.apache.org/confluence/display/KAFKA/Restful+API+Proposal
> >
> > I will read yours later this afternoon, we should work together.
> >
> > -Jonathan
> >
> >
> > On Fri, Aug 3, 2012 at 7:41 AM, David Arthur <[EMAIL PROTECTED]> wrote:
> >
> >> I'd like to tackle this project (assuming it hasn't been started yet).
> >>
> >> I wrote up some initial thoughts here: https://gist.github.com/3248179
> >>
> >> TLDR;  use Range header for specifying offsets, simple URIs like
> >> /kafka/topics/[topic]/[partition], use for a simple transport of bytes
> >> and/or represent the messages as some media type (text, json, xml)
> >>
> >> Feedback is most welcome (in the Gist or in this thread).
> >>
> >> Cheers!
> >>
> >> -David
>
>