Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # user >> async producer behavior if zk and/or kafka cluster goes away...


Copy link to this message
-
Re: async producer behavior if zk and/or kafka cluster goes away...
> By requiring use of a configured broker.list for each client, means that
> 1000's of deployed apps need to be updated any time the kafka cluster
> changes, no?  (Or am I not understanding?).

The advantage is that you can configure broker.list to point to a VIP, so you
can transparently change the brokers behind the VIP without having to
re-configure
the producers. On the other hand, if you ever had to make a similar
change to your
zookeeper cluster, it will be very operationally heavy since you will
have to make a
config change on each of your producers.

> You mention that auto-discovery of new brokers will still work, is that
> dependent on the existing configured broker.list set still being available
> also?

It depends on at least one broker in the cluster being alive. If none
of them are alive,
you have a much bigger problem to worry about.

Thanks,
Neha

On Tue, Nov 20, 2012 at 9:44 AM, Jason Rosenberg <[EMAIL PROTECTED]> wrote:
> Ok,
>
> So, I'm still wrapping my mind around this.  I liked being able to use zk
> for all clients, since it made it very easy to think about how to update
> the kafka cluster.  E.g. how to add new brokers, how to move them all to
> new hosts entirely, etc., without having to redeploy all the clients.  The
> new brokers will simply advertise their new location via zk, and all
> clients will pick it up.
>
> By requiring use of a configured broker.list for each client, means that
> 1000's of deployed apps need to be updated any time the kafka cluster
> changes, no?  (Or am I not understanding?).
>
> You mention that auto-discovery of new brokers will still work, is that
> dependent on the existing configured broker.list set still being available
> also?
>
> I can see though how this will greatly reduce the load on zookeeper.
>
> Jason
>
>
>
> On Tue, Nov 20, 2012 at 9:03 AM, Jun Rao <[EMAIL PROTECTED]> wrote:
>
>> Jason,
>>
>> Auto discovery of new brokers and rolling restart of brokers are still
>> supported in 0.8. It's just that most of the ZK related logic is moved to
>> the broker.
>>
>> There are 2 reasons why we want to remove zkclient from the client.
>>
>> 1. If the client goes to GC, it can cause zk session expiration and cause
>> churns in the client and extra load on the zk server.
>> 2. This simplifies the client code and makes the implementation of non-java
>> clients easier.
>>
>> In 0.8, we removed the zk dependency from the producer. Post 0.8, we plan
>> to see if we can do the same thing for the consumer (though more involved).
>> This shouldn't reduce any existing functionality of the client though. Feel
>> free to let us know if you still have concerns.
>>
>> Thanks,
>>
>> Jun
>>
>>
>>
>> On Tue, Nov 20, 2012 at 7:57 AM, Jason Rosenberg <[EMAIL PROTECTED]> wrote:
>>
>> > I checked out trunk.  I guess I assumed that included the latest 0.8.  Is
>> > that not right?  Am I just looking at 0.7.x+?
>> >
>> > Honestly, I don't think it would be a positive thing not to be able to
>> rely
>> > on zookeeper in producer code.  How does that affect the discovery of a
>> > kafka cluster under dynamic conditions?  We'd expect to have a much
>> higher
>> > SLA for the zookeeper cluster than for kafka.  We'd like to be able to
>> > freely do rolling restarts of the kafka cluster, etc.
>> >
>> > Also, it seems a bit asymetric to use zk for the kafka brokers and
>> > consumers, but not the producers.
>> >
>> > Jason
>> >
>> > On Mon, Nov 19, 2012 at 8:50 PM, Jay Kreps <[EMAIL PROTECTED]> wrote:
>> >
>> > > In 0.8 there is no way to use zookeeper from the producer and no
>> > connection
>> > > from the client. There isn't even a way to configure a zk connection.
>> Are
>> > > you sure you checked out the 0.8 branch?
>> > >
>> > > Check the code you've got:
>> > > *jkreps-mn:kafka-0.8 jkreps$ svn info*
>> > > *Path: .*
>> > > *URL: https://svn.apache.org/repos/asf/incubator/kafka/branches/0.8*
>> > > *Repository Root: https://svn.apache.org/repos/asf*
>> > >
>> > > The key is that it should have come from the URL kafka/branches/0.8.
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB