These are excellent points, thanks.

-Jay

On Thu, Jun 14, 2012 at 1:39 PM, Evan Chan <[EMAIL PROTECTED]> wrote:

> I would like to throw in a couple use cases:
>
>
>   - Allow the new consumer to reset its offset to either the current
>   largest or smallest.  This would be a great way to restart a process that
>   has fallen behind.  The only way I know how to do this today, with the
>   high-level consumer, is to delete the ZK nodes manually and restart the
>   consumer.
>   - Allow the consumer to reset its offset to some arbitrary value, and
>   then write that offset into ZK.    Kind of like the first case, but would
>   make rewinding/replays much easier.
>
> Modularity (the ability to layer the ZK infrastructure on top of the simple
> interface) would be great.
>
> thanks,
> Evan
>
>
> On Tue, Jun 12, 2012 at 9:59 AM, Jay Kreps <[EMAIL PROTECTED]> wrote:
>
> > This is a great summary Neha. It would be good to get people's feedback
> on
> > this since we don't want to keep breaking api and
> > protocol compatibility here, so the hope is to really get it right this
> > time now that we have really seen all the use cases and live with the
> > output for a while. I think the consumer design is a pretty hard protocol
> > and API design problem, so its fun to think about.
> >
> > If I were to summarize Neha's requirements list, I think there are three
> > high-level goals:
> >
> >   1. Simplify the consumer protocol to enable ease of development of
> >   consumer clients in other languages
> >   2. Try to replace the "simple consumer" and "high level consumer" with
> a
> >   single, general interface that has all the advantages of both.
> >   3. Support a bunch of use cases that either we didn't think of, or that
> >   weren't possible in the partitioning model of the pre-0.8 code base.
> >
> > -Jay
> >
> >
> > On Mon, Jun 11, 2012 at 4:52 PM, Neha Narkhede <[EMAIL PROTECTED]
> > >wrote:
> >
> > > Hi,
> > >
> > > Over the past few months, we've received quite a lot of feedback on the
> > > consumer side features and design. Some of them are improvements to the
> > > current consumer design and some are simply new feature/API requests. I
> > > have attempted to write up the requirements that I've heard on this
> wiki
> > -
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/Consumer+Client+Re-Design
> > >
> > > This would involve some significant changes to the consumer APIs, so we
> > > would like to collect feedback on the proposal from our community.
> Since
> > > the list of changes is not small, we would like to understand if some
> > > features are preferred over others, and more importantly, if some
> > features
> > > are not required at all.
> > >
> > > Since some part of this proposal is experimental and the consumer side
> > > changes are non-trivial, we would like this initiative to not interfere
> > > with the forthcoming replication release. However, it will be good to
> > have
> > > people from the community give this some thought and help out with the
> > > JIRAs if interested. One way of managing this project could be
> creating a
> > > separate branch from the kafka trunk and continue development on it.
> Once
> > > it is ready and in good shape, we can think about cutting another
> release
> > > (after 0.8) for the releasing the new consumer API. Do people have
> > > preferences/concerns regarding creating a separate branch for this
> > project
> > > ?
> > >
> > > Please feel free to start a discussion on this JIRA -
> > > https://issues.apache.org/jira/browse/KAFKA-364
> > >
> > > Thanks,
> > >
> > > Neha
> > >
> >
>
>
>
> --
> --
> *Evan Chan*
> Senior Software Engineer |
> [EMAIL PROTECTED] | (650) 996-4600
> www.ooyala.com | blog <http://www.ooyala.com/blog> |
> @ooyala<http://www.twitter.com/ooyala>
>

That's true - I think that's one of the major motivations of the consumer
re-design. Right now, the consumer implementation is very thick which makes
it difficult to maintain correct implementations across multiple languages.
It will be much easier to implement a consumer with the thinner logic - and
as you pointed out, many languages have pretty good bindings with native C
libraries so technically we would go pretty far with just a JVM and native
(C) implementation of the consumer logic.

Joel

On Mon, Jun 18, 2012 at 11:40 AM, Sybrandy, Casey <
[EMAIL PROTECTED]> wrote:

> Would porting the consumer/producer code to C be a good idea?  I say this
> because at least with most languages I know of, leveraging a C library is
> pretty easy.  This way, you would have to maintain only the C library and
> others can make/maintain wrappers for their languages.  Having to port to
> other languages is going to cause you to have a significant amount of
> maintenance if you change the protocol in the future.
>
> ________________________________________
> From: Neha Narkhede [[EMAIL PROTECTED]]
> Sent: Thursday, June 14, 2012 5:53 PM
> To: [EMAIL PROTECTED]
> Cc: [EMAIL PROTECTED]
> Subject: Re: Consumer re-design proposal
>
> Thanks for the feedback ! I moved it to
> https://issues.apache.org/jira/browse/KAFKA-364, so that we can keep track
> of these.
>
> -Neha
>
> On Thu, Jun 14, 2012 at 2:45 PM, Marcos Juarez <[EMAIL PROTECTED]> wrote:
>
> > Throwing a +1 on "Allow the consumer to reset its offset to some
> arbitrary
> > value, and then write that offset into ZK".
> >
> > We're currently running into a scenario where we would like to have 100%
> > reliability, and we're losing a few messages when a connection is broken,
> > but there were still a few messages in the OS TCP buffers. So, we're
> > planning on shifting the ZK offset by a few seconds "back in time" if we
> > detect a broker has gone down, to make sure all the messages will be
> > actually delivered to the end consumer when that broker comes back up,
> even
> > if there's a small amount of overlapping messages.
> >
> > Thanks,
> >
> > Marcos
> >
> >
> > On Jun 14, 2012, at 2:39 PM, Evan Chan wrote:
> >
> > > I would like to throw in a couple use cases:
> > >
> > >
> > >   - Allow the new consumer to reset its offset to either the current
> > >   largest or smallest.  This would be a great way to restart a process
> > that
> > >   has fallen behind.  The only way I know how to do this today, with
> the
> > >   high-level consumer, is to delete the ZK nodes manually and restart
> the
> > >   consumer.
> > >   - Allow the consumer to reset its offset to some arbitrary value, and
> > >   then write that offset into ZK.    Kind of like the first case, but
> > would
> > >   make rewinding/replays much easier.
> > >
> > > Modularity (the ability to layer the ZK infrastructure on top of the
> > simple
> > > interface) would be great.
> > >
> > > thanks,
> > > Evan
> > >
> > >
> > > On Tue, Jun 12, 2012 at 9:59 AM, Jay Kreps <[EMAIL PROTECTED]>
> wrote:
> > >
> > >> This is a great summary Neha. It would be good to get people's
> feedback
> > on
> > >> this since we don't want to keep breaking api and
> > >> protocol compatibility here, so the hope is to really get it right
> this
> > >> time now that we have really seen all the use cases and live with the
> > >> output for a while. I think the consumer design is a pretty hard
> > protocol
> > >> and API design problem, so its fun to think about.
> > >>
> > >> If I were to summarize Neha's requirements list, I think there are
> three
> > >> high-level goals:
> > >>
> > >>  1. Simplify the consumer protocol to enable ease of development of
> > >>  consumer clients in other languages
> > >>  2. Try to replace the "simple consumer" and "high level consumer"
> with
> > a
> > >>  single, general interface that has all the advantages of both.
> > >>  3. Support a bunch of use cases that either we didn't think of, or

This would definitely be good to have. As Joel says we are thinking
about re-factoring the consumer protocol to make it much thinner to help
enable non-java consumers much more easily (i.e. get rid of zookeeper and
other things).

There is a very brief write up on this here:
https://cwiki.apache.org/confluence/display/KAFKA/Central+Consumer+Coordination

Take a look and let us know what you think. It might make sense to wait
until this is done before adding consumer clients.

-Jay

On Mon, Jun 18, 2012 at 1:41 PM, Joel Koshy <[EMAIL PROTECTED]> wrote:

> That's true - I think that's one of the major motivations of the consumer
> re-design. Right now, the consumer implementation is very thick which makes
> it difficult to maintain correct implementations across multiple languages.
> It will be much easier to implement a consumer with the thinner logic - and
> as you pointed out, many languages have pretty good bindings with native C
> libraries so technically we would go pretty far with just a JVM and native
> (C) implementation of the consumer logic.
>
> Joel
>
> On Mon, Jun 18, 2012 at 11:40 AM, Sybrandy, Casey <
> [EMAIL PROTECTED]> wrote:
>
> > Would porting the consumer/producer code to C be a good idea?  I say this
> > because at least with most languages I know of, leveraging a C library is
> > pretty easy.  This way, you would have to maintain only the C library and
> > others can make/maintain wrappers for their languages.  Having to port to
> > other languages is going to cause you to have a significant amount of
> > maintenance if you change the protocol in the future.
> >
> > ________________________________________
> > From: Neha Narkhede [[EMAIL PROTECTED]]
> > Sent: Thursday, June 14, 2012 5:53 PM
> > To: [EMAIL PROTECTED]
> > Cc: [EMAIL PROTECTED]
> > Subject: Re: Consumer re-design proposal
> >
> > Thanks for the feedback ! I moved it to
> > https://issues.apache.org/jira/browse/KAFKA-364, so that we can keep
> track
> > of these.
> >
> > -Neha
> >
> > On Thu, Jun 14, 2012 at 2:45 PM, Marcos Juarez <[EMAIL PROTECTED]>
> wrote:
> >
> > > Throwing a +1 on "Allow the consumer to reset its offset to some
> > arbitrary
> > > value, and then write that offset into ZK".
> > >
> > > We're currently running into a scenario where we would like to have
> 100%
> > > reliability, and we're losing a few messages when a connection is
> broken,
> > > but there were still a few messages in the OS TCP buffers. So, we're
> > > planning on shifting the ZK offset by a few seconds "back in time" if
> we
> > > detect a broker has gone down, to make sure all the messages will be
> > > actually delivered to the end consumer when that broker comes back up,
> > even
> > > if there's a small amount of overlapping messages.
> > >
> > > Thanks,
> > >
> > > Marcos
> > >
> > >
> > > On Jun 14, 2012, at 2:39 PM, Evan Chan wrote:
> > >
> > > > I would like to throw in a couple use cases:
> > > >
> > > >
> > > >   - Allow the new consumer to reset its offset to either the current
> > > >   largest or smallest.  This would be a great way to restart a
> process
> > > that
> > > >   has fallen behind.  The only way I know how to do this today, with
> > the
> > > >   high-level consumer, is to delete the ZK nodes manually and restart
> > the
> > > >   consumer.
> > > >   - Allow the consumer to reset its offset to some arbitrary value,
> and
> > > >   then write that offset into ZK.    Kind of like the first case, but
> > > would
> > > >   make rewinding/replays much easier.
> > > >
> > > > Modularity (the ability to layer the ZK infrastructure on top of the
> > > simple
> > > > interface) would be great.
> > > >
> > > > thanks,
> > > > Evan
> > > >
> > > >
> > > > On Tue, Jun 12, 2012 at 9:59 AM, Jay Kreps <[EMAIL PROTECTED]>
> > wrote:
> > > >
> > > >> This is a great summary Neha. It would be good to get people's
> > feedback
> > > on
> > > >> this since we don't want to keep breaking api and
> > > >> protocol compatibility here, so the hope is to really get it right

On 06/18/2012 06:29 PM, Jay Kreps wrote:
> This would definitely be good to have. As Joel says we are thinking
> about re-factoring the consumer protocol to make it much thinner to help
> enable non-java consumers much more easily (i.e. get rid of zookeeper and
> other things).

This isn't removing ZK from Kafka, but rather having the clients only
interact with a broker side API, right?

+list

The sentiment makes sense.  I just know with the people I talked to the
(arguably deceptive) simplicity of the SimpleConsumer was a significant
selling point.

On 2012-06-19 01:18, Jay Kreps wrote:
> I think the hope was to have a single API and make each piece of
> functionality optional. E.g. you could have manual offset management OR you
> could have manual partition assignment OR both. You are right that if you
> disable everything you are left with something not much better than the
> current simple consumer except that it would support leadership changes (a
> la replication). But somehow having a single public API seems like a good
> practice just to have a single place where monitoring, testing, etc.
>
> -Jay
>
> On Mon, Jun 18, 2012 at 7:00 PM, Chris Burroughs
> <[EMAIL PROTECTED]>wrote:
>
>> On 06/12/2012 12:59 PM, Jay Kreps wrote:
>>>    2. Try to replace the "simple consumer" and "high level consumer"
>> with a
>>>    single, general interface that has all the advantages of both.
>>
>> I've read through the wiki pages but think I'm missing the forrest for
>> the trees.
>>
>> For a consumer that wants "Manual partition assignment" and "Manual
>> offset management", what does the proposed offer over the existing
>> SimpleConsumer?
>>
>

Dave,

Just to clarify. We are not removing ZK completely. The proposal is just
trying to see if we can remove the ZK dependency on the consumer client. ZK
will still be used at the broker.

Thanks,

Jun

On Tue, Jun 19, 2012 at 10:49 PM, Dave Barr <[EMAIL PROTECTED]> wrote:

> On Tue, Jun 19, 2012 at 10:09 AM, Neha Narkhede <[EMAIL PROTECTED]>
> wrote:
> > One of the goals of thinning the Kafka consumer client is removing the
> > zookeeper client from the consumer. Without this, Kafka consumer
> > client would depend on the stability of a zookeeper client.
>
> If there's a stability issue with the zookeeper client, then that
> should be addressed.
>
> ZK is a fine tool for service discovery and coordination.  It seems
> like any new system that forced me, as a consumer, to use yet another
> system to bootstrap and discover where my brokers are for a topic
> would be a step backward.
>
> I'm curious, why, specifically, is removing ZK a design goal
> (especially when it's such a core component of the broker)?  I think
> of other projects, like HBase, which seem to have no issue with using
> ZK in their client.
>
> --Dave
>

Fwiw I think this is the right move. We don't use ZK in our Kafka installation.

One reason is that we do not want or need the complexity of having the
broker distribution in the clients.

Moving towards this design in the core would help our installation be
more "standard".

On Jun 20, 2012, at 7:48 AM, Jun Rao <[EMAIL PROTECTED]> wrote:

> Dave,
>
> Just to clarify. We are not removing ZK completely. The proposal is just
> trying to see if we can remove the ZK dependency on the consumer client. ZK
> will still be used at the broker.
>
> Thanks,
>
> Jun
>
> On Tue, Jun 19, 2012 at 10:49 PM, Dave Barr <[EMAIL PROTECTED]> wrote:
>
>> On Tue, Jun 19, 2012 at 10:09 AM, Neha Narkhede <[EMAIL PROTECTED]>
>> wrote:
>>> One of the goals of thinning the Kafka consumer client is removing the
>>> zookeeper client from the consumer. Without this, Kafka consumer
>>> client would depend on the stability of a zookeeper client.
>>
>> If there's a stability issue with the zookeeper client, then that
>> should be addressed.
>>
>> ZK is a fine tool for service discovery and coordination.  It seems
>> like any new system that forced me, as a consumer, to use yet another
>> system to bootstrap and discover where my brokers are for a topic
>> would be a step backward.
>>
>> I'm curious, why, specifically, is removing ZK a design goal
>> (especially when it's such a core component of the broker)?  I think
>> of other projects, like HBase, which seem to have no issue with using
>> ZK in their client.
>>
>> --Dave
>>

How would you accomplish automatic rebalancing without ZK?

On Wed, Jun 20, 2012 at 8:25 AM, Taylor Gautier <[EMAIL PROTECTED]> wrote:

> Fwiw I think this is the right move. We don't use ZK in our Kafka
> installation.
>
> One reason is that we do not want or need the complexity of having the
> broker distribution in the clients.
>
> Moving towards this design in the core would help our installation be
> more "standard".
>
>
>
> On Jun 20, 2012, at 7:48 AM, Jun Rao <[EMAIL PROTECTED]> wrote:
>
> > Dave,
> >
> > Just to clarify. We are not removing ZK completely. The proposal is just
> > trying to see if we can remove the ZK dependency on the consumer client.
> ZK
> > will still be used at the broker.
> >
> > Thanks,
> >
> > Jun
> >
> > On Tue, Jun 19, 2012 at 10:49 PM, Dave Barr <[EMAIL PROTECTED]> wrote:
> >
> >> On Tue, Jun 19, 2012 at 10:09 AM, Neha Narkhede <
> [EMAIL PROTECTED]>
> >> wrote:
> >>> One of the goals of thinning the Kafka consumer client is removing the
> >>> zookeeper client from the consumer. Without this, Kafka consumer
> >>> client would depend on the stability of a zookeeper client.
> >>
> >> If there's a stability issue with the zookeeper client, then that
> >> should be addressed.
> >>
> >> ZK is a fine tool for service discovery and coordination.  It seems
> >> like any new system that forced me, as a consumer, to use yet another
> >> system to bootstrap and discover where my brokers are for a topic
> >> would be a step backward.
> >>
> >> I'm curious, why, specifically, is removing ZK a design goal
> >> (especially when it's such a core component of the broker)?  I think
> >> of other projects, like HBase, which seem to have no issue with using
> >> ZK in their client.
> >>
> >> --Dave
> >>
>

--
--
*Evan Chan*
Senior Software Engineer |
[EMAIL PROTECTED] | (650) 996-4600
www.ooyala.com | blog <http://www.ooyala.com/blog> |
@ooyala<http://www.twitter.com/ooyala>

The proposal is to move the rebalancing logic out of the consumer to
the broker side. The consumer client merely connects to the consumer
co-ordinator (on the Kafka brokers) and fetches data from the assigned
set of partitions. We have a high level design here -
https://cwiki.apache.org/confluence/display/KAFKA/Central+Consumer+Coordination#CentralConsumerCoordination-ProposalII

Thanks,
Neha

On Wed, Jun 20, 2012 at 9:36 AM, Evan Chan <[EMAIL PROTECTED]> wrote:
> How would you accomplish automatic rebalancing without ZK?
>
> On Wed, Jun 20, 2012 at 8:25 AM, Taylor Gautier <[EMAIL PROTECTED]> wrote:
>
>> Fwiw I think this is the right move. We don't use ZK in our Kafka
>> installation.
>>
>> One reason is that we do not want or need the complexity of having the
>> broker distribution in the clients.
>>
>> Moving towards this design in the core would help our installation be
>> more "standard".
>>
>>
>>
>> On Jun 20, 2012, at 7:48 AM, Jun Rao <[EMAIL PROTECTED]> wrote:
>>
>> > Dave,
>> >
>> > Just to clarify. We are not removing ZK completely. The proposal is just
>> > trying to see if we can remove the ZK dependency on the consumer client.
>> ZK
>> > will still be used at the broker.
>> >
>> > Thanks,
>> >
>> > Jun
>> >
>> > On Tue, Jun 19, 2012 at 10:49 PM, Dave Barr <[EMAIL PROTECTED]> wrote:
>> >
>> >> On Tue, Jun 19, 2012 at 10:09 AM, Neha Narkhede <
>> [EMAIL PROTECTED]>
>> >> wrote:
>> >>> One of the goals of thinning the Kafka consumer client is removing the
>> >>> zookeeper client from the consumer. Without this, Kafka consumer
>> >>> client would depend on the stability of a zookeeper client.
>> >>
>> >> If there's a stability issue with the zookeeper client, then that
>> >> should be addressed.
>> >>
>> >> ZK is a fine tool for service discovery and coordination.  It seems
>> >> like any new system that forced me, as a consumer, to use yet another
>> >> system to bootstrap and discover where my brokers are for a topic
>> >> would be a step backward.
>> >>
>> >> I'm curious, why, specifically, is removing ZK a design goal
>> >> (especially when it's such a core component of the broker)?  I think
>> >> of other projects, like HBase, which seem to have no issue with using
>> >> ZK in their client.
>> >>
>> >> --Dave
>> >>
>>
>
>
>
> --
> --
> *Evan Chan*
> Senior Software Engineer |
> [EMAIL PROTECTED] | (650) 996-4600
> www.ooyala.com | blog <http://www.ooyala.com/blog> |
> @ooyala<http://www.twitter.com/ooyala>

Yes, which is why I asked how discovery will work.

Proposal II reads a bit handwavy.

"On startup, the consumer is informed with a list of the brokers."

>From where?  From whom?  I have to provide some mechanism now to feed
this config to my consumers?  What happens when this changes?

Say I start up my producers and consumers with a static list of hosts
A,B,C.  Over time, I add host D, E, F.  D becomes the eventual new
coordinator.  But the clients have an in-memory config which knows
only of A, B, and C.  Now host D dies.  What happens (especially to
clients who have never been told about/connected to E or F)?

Also is this proposal going to address the ability to drain a broker?
Say I have a broker with old data that I want to keep around so you
can reconsume old data, but I don't want anyone producing to it
anymore.

--Dave
On Wed, Jun 20, 2012 at 7:48 AM, Jun Rao <[EMAIL PROTECTED]> wrote:
> Dave,
>
> Just to clarify. We are not removing ZK completely. The proposal is just
> trying to see if we can remove the ZK dependency on the consumer client. ZK
> will still be used at the broker.
>
> Thanks,
>
> Jun
>
> On Tue, Jun 19, 2012 at 10:49 PM, Dave Barr <[EMAIL PROTECTED]> wrote:
>
>> On Tue, Jun 19, 2012 at 10:09 AM, Neha Narkhede <[EMAIL PROTECTED]>
>> wrote:
>> > One of the goals of thinning the Kafka consumer client is removing the
>> > zookeeper client from the consumer. Without this, Kafka consumer
>> > client would depend on the stability of a zookeeper client.
>>
>> If there's a stability issue with the zookeeper client, then that
>> should be addressed.
>>
>> ZK is a fine tool for service discovery and coordination.  It seems
>> like any new system that forced me, as a consumer, to use yet another
>> system to bootstrap and discover where my brokers are for a topic
>> would be a step backward.
>>
>> I'm curious, why, specifically, is removing ZK a design goal
>> (especially when it's such a core component of the broker)?  I think
>> of other projects, like HBase, which seem to have no issue with using
>> ZK in their client.
>>
>> --Dave
>>

Dave,

> Say I start up my producers and consumers with a static list of hosts
> A,B,C.  Over time, I add host D, E, F.  D becomes the eventual new
> coordinator.  But the clients have an in-memory config which knows
> only of A, B, and C.  Now host D dies.  What happens (especially to
> clients who have never been told about/connected to E or F)?

The consumers would need only one of the brokers in its config to be
alive. It talks to one of the live brokers in the boostrap brokers
list and finds out who the co-ordinator is, and then communicates with
the co-ordinator.
So, in your example, unless you take A, B AND C offline, it shouldn't
be a problem. And if you need to do that, you can rolling restart the
consumers. Maybe there is another easier way to solve the bootstrap
problem,
this is just one of the solutions.

>> > Also is this proposal going to address the ability to drain a broker?
> Say I have a broker with old data that I want to keep around so you
> can reconsume old data, but I don't want anyone producing to it
> anymore.

This proposal is only dealing with Consumer API/protocol redesign.
What you want is the ability to decommission a broker gracefully. I
suggest you add your comments/requirements to this JIRA -
https://issues.apache.org/jira/browse/KAFKA-155

Also, how about moving your suggestions/comments to the JIRA/wiki ?
Email is not quite the easiest way to keep track of this discussion.

Thanks,
Neha

On Wed, Jun 20, 2012 at 12:11 PM, Dave Barr <[EMAIL PROTECTED]> wrote:
> Yes, which is why I asked how discovery will work.
>
> Proposal II reads a bit handwavy.
>
> "On startup, the consumer is informed with a list of the brokers."
>
> From where?  From whom?  I have to provide some mechanism now to feed
> this config to my consumers?  What happens when this changes?
>
> Say I start up my producers and consumers with a static list of hosts
> A,B,C.  Over time, I add host D, E, F.  D becomes the eventual new
> coordinator.  But the clients have an in-memory config which knows
> only of A, B, and C.  Now host D dies.  What happens (especially to
> clients who have never been told about/connected to E or F)?
>
> Also is this proposal going to address the ability to drain a broker?
> Say I have a broker with old data that I want to keep around so you
> can reconsume old data, but I don't want anyone producing to it
> anymore.
>
> --Dave
> On Wed, Jun 20, 2012 at 7:48 AM, Jun Rao <[EMAIL PROTECTED]> wrote:
>> Dave,
>>
>> Just to clarify. We are not removing ZK completely. The proposal is just
>> trying to see if we can remove the ZK dependency on the consumer client. ZK
>> will still be used at the broker.
>>
>> Thanks,
>>
>> Jun
>>
>> On Tue, Jun 19, 2012 at 10:49 PM, Dave Barr <[EMAIL PROTECTED]> wrote:
>>
>>> On Tue, Jun 19, 2012 at 10:09 AM, Neha Narkhede <[EMAIL PROTECTED]>
>>> wrote:
>>> > One of the goals of thinning the Kafka consumer client is removing the
>>> > zookeeper client from the consumer. Without this, Kafka consumer
>>> > client would depend on the stability of a zookeeper client.
>>>
>>> If there's a stability issue with the zookeeper client, then that
>>> should be addressed.
>>>
>>> ZK is a fine tool for service discovery and coordination.  It seems
>>> like any new system that forced me, as a consumer, to use yet another
>>> system to bootstrap and discover where my brokers are for a topic
>>> would be a step backward.
>>>
>>> I'm curious, why, specifically, is removing ZK a design goal
>>> (especially when it's such a core component of the broker)?  I think
>>> of other projects, like HBase, which seem to have no issue with using
>>> ZK in their client.
>>>
>>> --Dave
>>>

On 06/20/2012 11:25 AM, Taylor Gautier wrote:
> Fwiw I think this is the right move. We don't use ZK in our Kafka installation.

How do the consumers know who the brokers are?  List in a config file?