Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka, mail # user - async producer behavior if zk and/or kafka cluster goes away...


Copy link to this message
-
Re: async producer behavior if zk and/or kafka cluster goes away...
Jason Rosenberg 2012-11-20, 17:44
Ok,

So, I'm still wrapping my mind around this.  I liked being able to use zk
for all clients, since it made it very easy to think about how to update
the kafka cluster.  E.g. how to add new brokers, how to move them all to
new hosts entirely, etc., without having to redeploy all the clients.  The
new brokers will simply advertise their new location via zk, and all
clients will pick it up.

By requiring use of a configured broker.list for each client, means that
1000's of deployed apps need to be updated any time the kafka cluster
changes, no?  (Or am I not understanding?).

You mention that auto-discovery of new brokers will still work, is that
dependent on the existing configured broker.list set still being available
also?

I can see though how this will greatly reduce the load on zookeeper.

Jason

On Tue, Nov 20, 2012 at 9:03 AM, Jun Rao <[EMAIL PROTECTED]> wrote:

> Jason,
>
> Auto discovery of new brokers and rolling restart of brokers are still
> supported in 0.8. It's just that most of the ZK related logic is moved to
> the broker.
>
> There are 2 reasons why we want to remove zkclient from the client.
>
> 1. If the client goes to GC, it can cause zk session expiration and cause
> churns in the client and extra load on the zk server.
> 2. This simplifies the client code and makes the implementation of non-java
> clients easier.
>
> In 0.8, we removed the zk dependency from the producer. Post 0.8, we plan
> to see if we can do the same thing for the consumer (though more involved).
> This shouldn't reduce any existing functionality of the client though. Feel
> free to let us know if you still have concerns.
>
> Thanks,
>
> Jun
>
>
>
> On Tue, Nov 20, 2012 at 7:57 AM, Jason Rosenberg <[EMAIL PROTECTED]> wrote:
>
> > I checked out trunk.  I guess I assumed that included the latest 0.8.  Is
> > that not right?  Am I just looking at 0.7.x+?
> >
> > Honestly, I don't think it would be a positive thing not to be able to
> rely
> > on zookeeper in producer code.  How does that affect the discovery of a
> > kafka cluster under dynamic conditions?  We'd expect to have a much
> higher
> > SLA for the zookeeper cluster than for kafka.  We'd like to be able to
> > freely do rolling restarts of the kafka cluster, etc.
> >
> > Also, it seems a bit asymetric to use zk for the kafka brokers and
> > consumers, but not the producers.
> >
> > Jason
> >
> > On Mon, Nov 19, 2012 at 8:50 PM, Jay Kreps <[EMAIL PROTECTED]> wrote:
> >
> > > In 0.8 there is no way to use zookeeper from the producer and no
> > connection
> > > from the client. There isn't even a way to configure a zk connection.
> Are
> > > you sure you checked out the 0.8 branch?
> > >
> > > Check the code you've got:
> > > *jkreps-mn:kafka-0.8 jkreps$ svn info*
> > > *Path: .*
> > > *URL: https://svn.apache.org/repos/asf/incubator/kafka/branches/0.8*
> > > *Repository Root: https://svn.apache.org/repos/asf*
> > >
> > > The key is that it should have come from the URL kafka/branches/0.8.
> > >
> > > -Jay
> > >
> > >
> > > On Mon, Nov 19, 2012 at 3:30 PM, Jason Rosenberg <[EMAIL PROTECTED]>
> > wrote:
> > >
> > > > Regarding the poducer/zk connection:  if I am using zk to discover
> the
> > > > kafka cluster, doesn't the producer get updates if zk's knowledge of
> > the
> > > > cluster changes?  Or does it only reconsult zk if the particular
> kafka
> > > node
> > > > it was "getting metadata" from goes away?  Should I not be using a
> > > > "zk.connect" but instead a "broker.list" when using a producer (that
> > > would
> > > > seem restrictive)?  What I've noticed is that the instant the zk
> server
> > > is
> > > > taken down, my producer immediately starts logging connection errors
> to
> > > zk,
> > > > every second, and never stops this logging until zk comes back.  So
> it
> > > > certainly feels like the producer is attempting to maintain a direct
> > > > connection to zk.  I suppose I expected it to try for the connection
> > > > timeout period (e.g. 6000ms), and then give up, until the next send