Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka, mail # dev - Integration with AMQP

Copy link to this message
Re: Integration with AMQP
Milind Parikh 2012-10-03, 00:09
I'm not exactly sure about why talking the same at the wire level is an
explicit goal; if raw blazing speed is also equally important.

Removing the wireformat from Kafka could be done through abstraction; but
that would incur reinterpretation  costs(talking native Kafka) and take a
performance hit.

I could make a similar argument over the second goal as well. It is not
apparent that solving ALL problems through abstraction and then universally
accepting a performance hit is that ideal. It may make purchasing/acquiring
easier; but by adhering to the lowest common denominator.

Wouldn't it be better to make explicit tradeoffs for one-offs based on
specific needs? i.e. if your architecture doesnt require zookeeper in Kafka
for coordination, reduce the complexity. Don't force complexity in all

In other words, Kafka is different enough from other messaging systems that
to enforce a common contract (aka amqp) ,without incurring a significant
performance hit, would be very challenging.
 On Oct 2, 2012 3:22 PM, "William Henry" <[EMAIL PROTECTED]> wrote:

> ----- Original Message -----
> > I looked into AMQP when I was first starting Kafka work. I see the
> > crux of
> > the issue as this: if you have a bunch of systems that essentially
> > expose
> > the same functionality there is value in standardizing the protocol
> > by
> > which they are accessed to help decouple interface from
> > implementation. Of
> > course I think it is better still to end up with a single good
> > implementation (e.g. Linux rather than Posix). But invariably the
> > protocol
> > dictates the feature set, which dictates the implementation, and so
> > this
> > only really works if the systems have the same feature set and
> > similar
> > enough implementations. This becomes true in a domain over time as
> > people
> > learn the best way to build that kind of system, and all the systems
> > converge to that.
> +1
> >
> > The reason we have not been pursuing this is that I think the set of
> > functionality we are aiming for is a little different than what most
> > message brokers have. Basically the idea we have is to attempt to
> > re-imagine "messaging" or asynchronous processing infrastructure as a
> > distributed, replicated, partitioned "commit log". This is different
> > enough
> > from what other system do that attempting to support a standardized
> > protocol is unlikely to work out well. For example, the consumer
> > balancing
> > we do is not modeled in AMQP, and there are many AMQP features that
> > Kafka
> > doesn't have.
> I need to understand your consumer balancing a bit more but AMQP is
> designed not to be another MOM like traditional broker based messaging
> systems, though it does support that model.
> I like to explain the goals of AMQP to be threefold (some may argue
> differently):
> 1) A Standard wire protocol for interoperability.  i.e. have all messaging
> systems speak the same on the wire.
> 2) Handle all messaging use cases well - i.e. not just asynch, not just
> fanout, not just pub/sub but instead do it all so that AMQP is applicable
> to all use cases. Let's not have a "we do AMQP everywhere except X because
> it does do X very well.
> 3) Must be fast. Even if it does 1 and 2 very well it will not be adopted
> by a wide range of applications.
> So if by consumer balancing you mean multiple consumers feeding off a
> particular address/source/publisher/producer etc. then AMQP does manage
> that model.
> >
> > Basically I don't really see other messaging systems as being fully
> > formed
> > distributed systems that acts as a *cluster* (rather than an ensemble
> > of
> > brokers).
> This is exactly what we in the Qpid community are working towards right
> now.  I think AMQP as a protocol under Kafka and exploiting Kafka's
> framework is a great idea.
> Please look at the new Qpid/Proton work and some of Ted Ross's (cc-ed)
> router work.
> > Conceptually when people program to, say, HDFS, you largely