Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # user >> async producer behavior if zk and/or kafka cluster goes away...


Copy link to this message
-
Re: async producer behavior if zk and/or kafka cluster goes away...
> By requiring use of a configured broker.list for each client, means that
> 1000's of deployed apps need to be updated any time the kafka cluster
> changes, no?  (Or am I not understanding?).

The advantage is that you can configure broker.list to point to a VIP, so you
can transparently change the brokers behind the VIP without having to
re-configure
the producers. On the other hand, if you ever had to make a similar
change to your
zookeeper cluster, it will be very operationally heavy since you will
have to make a
config change on each of your producers.

> You mention that auto-discovery of new brokers will still work, is that
> dependent on the existing configured broker.list set still being available
> also?

It depends on at least one broker in the cluster being alive. If none
of them are alive,
you have a much bigger problem to worry about.

Thanks,
Neha

On Tue, Nov 20, 2012 at 9:44 AM, Jason Rosenberg <[EMAIL PROTECTED]> wrote:
> Ok,
>
> So, I'm still wrapping my mind around this.  I liked being able to use zk
> for all clients, since it made it very easy to think about how to update
> the kafka cluster.  E.g. how to add new brokers, how to move them all to
> new hosts entirely, etc., without having to redeploy all the clients.  The
> new brokers will simply advertise their new location via zk, and all
> clients will pick it up.
>
> By requiring use of a configured broker.list for each client, means that
> 1000's of deployed apps need to be updated any time the kafka cluster
> changes, no?  (Or am I not understanding?).
>
> You mention that auto-discovery of new brokers will still work, is that
> dependent on the existing configured broker.list set still being available
> also?
>
> I can see though how this will greatly reduce the load on zookeeper.
>
> Jason
>
>
>
> On Tue, Nov 20, 2012 at 9:03 AM, Jun Rao <[EMAIL PROTECTED]> wrote:
>
>> Jason,
>>
>> Auto discovery of new brokers and rolling restart of brokers are still
>> supported in 0.8. It's just that most of the ZK related logic is moved to
>> the broker.
>>
>> There are 2 reasons why we want to remove zkclient from the client.
>>
>> 1. If the client goes to GC, it can cause zk session expiration and cause
>> churns in the client and extra load on the zk server.
>> 2. This simplifies the client code and makes the implementation of non-java
>> clients easier.
>>
>> In 0.8, we removed the zk dependency from the producer. Post 0.8, we plan
>> to see if we can do the same thing for the consumer (though more involved).
>> This shouldn't reduce any existing functionality of the client though. Feel
>> free to let us know if you still have concerns.
>>
>> Thanks,
>>
>> Jun
>>
>>
>>
>> On Tue, Nov 20, 2012 at 7:57 AM, Jason Rosenberg <[EMAIL PROTECTED]> wrote:
>>
>> > I checked out trunk.  I guess I assumed that included the latest 0.8.  Is
>> > that not right?  Am I just looking at 0.7.x+?
>> >
>> > Honestly, I don't think it would be a positive thing not to be able to
>> rely
>> > on zookeeper in producer code.  How does that affect the discovery of a
>> > kafka cluster under dynamic conditions?  We'd expect to have a much
>> higher
>> > SLA for the zookeeper cluster than for kafka.  We'd like to be able to
>> > freely do rolling restarts of the kafka cluster, etc.
>> >
>> > Also, it seems a bit asymetric to use zk for the kafka brokers and
>> > consumers, but not the producers.
>> >
>> > Jason
>> >
>> > On Mon, Nov 19, 2012 at 8:50 PM, Jay Kreps <[EMAIL PROTECTED]> wrote:
>> >
>> > > In 0.8 there is no way to use zookeeper from the producer and no
>> > connection
>> > > from the client. There isn't even a way to configure a zk connection.
>> Are
>> > > you sure you checked out the 0.8 branch?
>> > >
>> > > Check the code you've got:
>> > > *jkreps-mn:kafka-0.8 jkreps$ svn info*
>> > > *Path: .*
>> > > *URL: https://svn.apache.org/repos/asf/incubator/kafka/branches/0.8*
>> > > *Repository Root: https://svn.apache.org/repos/asf*
>> > >
>> > > The key is that it should have come from the URL kafka/branches/0.8.