Hi,

I'm wondering if there's a reason kafka currently seems to depend on
zookeeper 3.3.4.  While the current stable version for zookeeper is 3.4.4.

Are there known issues with upgrading to the newer version of zookeeper?

Jason

The ZooKeeper API has been stable for a while, I do not suspect there would be any problems between 3.3.4 and 3.4.x. Personally, I have run Kafka 0.7.x with ZooKeeper 3.4.3 with no problems

-David

On Nov 13, 2012, at 2:30 AM, Jason Rosenberg wrote:

> Hi,
>
> I'm wondering if there's a reason kafka currently seems to depend on
> zookeeper 3.3.4.  While the current stable version for zookeeper is 3.4.4.
>
> Are there known issues with upgrading to the newer version of zookeeper?
>
> Jason

Jason,

We are just not sure about the stability of ZK 3.4.4. Does anybody know? In
fact, some of the features in ZK 3.4.x (like the multi operation) can
improve the availability of Kafka in 0.8. So, we'd like to move to ZK 3.4.4
if it's been widely used and is stable.

Thanks,

Jun

On Mon, Nov 12, 2012 at 11:30 PM, Jason Rosenberg <[EMAIL PROTECTED]> wrote:

> Hi,
>
> I'm wondering if there's a reason kafka currently seems to depend on
> zookeeper 3.3.4.  While the current stable version for zookeeper is 3.4.4.
>
> Are there known issues with upgrading to the newer version of zookeeper?
>
> Jason
>

I've been observing the zookeeper mailing list for 3.4.x and haven't
found it to be particularly stable. There are still serious issues
being reported like leader election failures, ephemeral node deletion.
I'm sure they are being fixed and there are many early adopters. But
my guess is that we should probably release 0.8 on a proven stable
version and look into 3.4.x for Kafka 0.8.1.

Thanks,
Neha

On Tue, Nov 13, 2012 at 6:57 AM, Jun Rao <[EMAIL PROTECTED]> wrote:
> Jason,
>
> We are just not sure about the stability of ZK 3.4.4. Does anybody know? In
> fact, some of the features in ZK 3.4.x (like the multi operation) can
> improve the availability of Kafka in 0.8. So, we'd like to move to ZK 3.4.4
> if it's been widely used and is stable.
>
> Thanks,
>
> Jun
>
> On Mon, Nov 12, 2012 at 11:30 PM, Jason Rosenberg <[EMAIL PROTECTED]> wrote:
>
>> Hi,
>>
>> I'm wondering if there's a reason kafka currently seems to depend on
>> zookeeper 3.3.4.  While the current stable version for zookeeper is 3.4.4.
>>
>> Are there known issues with upgrading to the newer version of zookeeper?
>>
>> Jason
>>

Ok,

So, there are known api changes that would preclude use of 3.4.x with
kafka, it sounds like?

Is there a notion of a supported version of zookeeper for each kafka
release?

How about upgrading the recommended version for 0.8 at least to the latest
3.3.x (3.3.6)?

Jason

On Tue, Nov 13, 2012 at 8:50 AM, Neha Narkhede <[EMAIL PROTECTED]>wrote:

> I've been observing the zookeeper mailing list for 3.4.x and haven't
> found it to be particularly stable. There are still serious issues
> being reported like leader election failures, ephemeral node deletion.
> I'm sure they are being fixed and there are many early adopters. But
> my guess is that we should probably release 0.8 on a proven stable
> version and look into 3.4.x for Kafka 0.8.1.
>
> Thanks,
> Neha
>
> On Tue, Nov 13, 2012 at 6:57 AM, Jun Rao <[EMAIL PROTECTED]> wrote:
> > Jason,
> >
> > We are just not sure about the stability of ZK 3.4.4. Does anybody know?
> In
> > fact, some of the features in ZK 3.4.x (like the multi operation) can
> > improve the availability of Kafka in 0.8. So, we'd like to move to ZK
> 3.4.4
> > if it's been widely used and is stable.
> >
> > Thanks,
> >
> > Jun
> >
> > On Mon, Nov 12, 2012 at 11:30 PM, Jason Rosenberg <[EMAIL PROTECTED]>
> wrote:
> >
> >> Hi,
> >>
> >> I'm wondering if there's a reason kafka currently seems to depend on
> >> zookeeper 3.3.4.  While the current stable version for zookeeper is
> 3.4.4.
> >>
> >> Are there known issues with upgrading to the newer version of zookeeper?
> >>
> >> Jason
> >>
>

> So, there are known api changes that would preclude use of 3.4.x with
> kafka, it sounds like?

Not really. Zookeeper 3.4.x is backwards compatible. Kafka 0.8 should
still work with any of the stable zookeeper versions out there. Once
ZK 3.4.x is stable, Kafka 0.8 would certainly benefit from the multi
APIs, since instead of doing one write per partition, we can do one
write for all partitions.

> Is there a notion of a supported version of zookeeper for each kafka
> release?

Not formally, though this sounds like a good idea. Implicitly, we
consider the zk version we release with as stable.

> How about upgrading the recommended version for 0.8 at least to the latest
> 3.3.x (3.3.6)?

Yes, do you have an idea about any particular bugs that were fixed
post 3.3.4 which could be useful ?

Thanks,
Neha

On Tue, Nov 13, 2012 at 9:54 AM, Jason Rosenberg <[EMAIL PROTECTED]> wrote:
> Ok,
>
> So, there are known api changes that would preclude use of 3.4.x with
> kafka, it sounds like?
>
> Is there a notion of a supported version of zookeeper for each kafka
> release?
>
> How about upgrading the recommended version for 0.8 at least to the latest
> 3.3.x (3.3.6)?
>
> Jason
>
>
>
> On Tue, Nov 13, 2012 at 8:50 AM, Neha Narkhede <[EMAIL PROTECTED]>wrote:
>
>> I've been observing the zookeeper mailing list for 3.4.x and haven't
>> found it to be particularly stable. There are still serious issues
>> being reported like leader election failures, ephemeral node deletion.
>> I'm sure they are being fixed and there are many early adopters. But
>> my guess is that we should probably release 0.8 on a proven stable
>> version and look into 3.4.x for Kafka 0.8.1.
>>
>> Thanks,
>> Neha
>>
>> On Tue, Nov 13, 2012 at 6:57 AM, Jun Rao <[EMAIL PROTECTED]> wrote:
>> > Jason,
>> >
>> > We are just not sure about the stability of ZK 3.4.4. Does anybody know?
>> In
>> > fact, some of the features in ZK 3.4.x (like the multi operation) can
>> > improve the availability of Kafka in 0.8. So, we'd like to move to ZK
>> 3.4.4
>> > if it's been widely used and is stable.
>> >
>> > Thanks,
>> >
>> > Jun
>> >
>> > On Mon, Nov 12, 2012 at 11:30 PM, Jason Rosenberg <[EMAIL PROTECTED]>
>> wrote:
>> >
>> >> Hi,
>> >>
>> >> I'm wondering if there's a reason kafka currently seems to depend on
>> >> zookeeper 3.3.4.  While the current stable version for zookeeper is
>> 3.4.4.
>> >>
>> >> Are there known issues with upgrading to the newer version of zookeeper?
>> >>
>> >> Jason
>> >>
>>

According to the zookeeper home page, the current stable releases are
"3.3.6" and "3.4.4".

There seem to be a few interesting bug fixes in the release notes for 3.3.5
& 3.3.6:
http://zookeeper.apache.org/doc/r3.3.5/releasenotes.html#changes
http://zookeeper.apache.org/doc/r3.3.6/releasenotes.html#changes

Notably fixes for:
  leaking file handles
  data inconsistencies and unexpired ephemeral nodes after cluster restart
  shutdown synchronization

But, I have not been using kafka long enough to see these issues myself.

On Tue, Nov 13, 2012 at 3:29 PM, Neha Narkhede <[EMAIL PROTECTED]>wrote:

> > So, there are known api changes that would preclude use of 3.4.x with
> > kafka, it sounds like?
>
> Not really. Zookeeper 3.4.x is backwards compatible. Kafka 0.8 should
> still work with any of the stable zookeeper versions out there. Once
> ZK 3.4.x is stable, Kafka 0.8 would certainly benefit from the multi
> APIs, since instead of doing one write per partition, we can do one
> write for all partitions.
>
> > Is there a notion of a supported version of zookeeper for each kafka
> > release?
>
> Not formally, though this sounds like a good idea. Implicitly, we
> consider the zk version we release with as stable.
>
> > How about upgrading the recommended version for 0.8 at least to the
> latest
> > 3.3.x (3.3.6)?
>
> Yes, do you have an idea about any particular bugs that were fixed
> post 3.3.4 which could be useful ?
>
> Thanks,
> Neha
>
> On Tue, Nov 13, 2012 at 9:54 AM, Jason Rosenberg <[EMAIL PROTECTED]> wrote:
> > Ok,
> >
> > So, there are known api changes that would preclude use of 3.4.x with
> > kafka, it sounds like?
> >
> > Is there a notion of a supported version of zookeeper for each kafka
> > release?
> >
> > How about upgrading the recommended version for 0.8 at least to the
> latest
> > 3.3.x (3.3.6)?
> >
> > Jason
> >
> >
> >
> > On Tue, Nov 13, 2012 at 8:50 AM, Neha Narkhede <[EMAIL PROTECTED]
> >wrote:
> >
> >> I've been observing the zookeeper mailing list for 3.4.x and haven't
> >> found it to be particularly stable. There are still serious issues
> >> being reported like leader election failures, ephemeral node deletion.
> >> I'm sure they are being fixed and there are many early adopters. But
> >> my guess is that we should probably release 0.8 on a proven stable
> >> version and look into 3.4.x for Kafka 0.8.1.
> >>
> >> Thanks,
> >> Neha
> >>
> >> On Tue, Nov 13, 2012 at 6:57 AM, Jun Rao <[EMAIL PROTECTED]> wrote:
> >> > Jason,
> >> >
> >> > We are just not sure about the stability of ZK 3.4.4. Does anybody
> know?
> >> In
> >> > fact, some of the features in ZK 3.4.x (like the multi operation) can
> >> > improve the availability of Kafka in 0.8. So, we'd like to move to ZK
> >> 3.4.4
> >> > if it's been widely used and is stable.
> >> >
> >> > Thanks,
> >> >
> >> > Jun
> >> >
> >> > On Mon, Nov 12, 2012 at 11:30 PM, Jason Rosenberg <[EMAIL PROTECTED]>
> >> wrote:
> >> >
> >> >> Hi,
> >> >>
> >> >> I'm wondering if there's a reason kafka currently seems to depend on
> >> >> zookeeper 3.3.4.  While the current stable version for zookeeper is
> >> 3.4.4.
> >> >>
> >> >> Are there known issues with upgrading to the newer version of
> zookeeper?
> >> >>
> >> >> Jason
> >> >>
> >>
>

Thanks for pulling up that list Jason ! I think that makes sense and
we can experiment with ZK 3.3.5. Not sure how widely 3.3.6 is tested
though.

Regarding the 3.4.x stability, the fact that serious issues are still
being reported makes me uncomfortable -

https://issues.apache.org/jira/browse/ZOOKEEPER-1496
https://issues.apache.org/jira/browse/ZOOKEEPER-1419

Thanks,
Neha

On Tue, Nov 13, 2012 at 4:36 PM, Jason Rosenberg <[EMAIL PROTECTED]> wrote:
> According to the zookeeper home page, the current stable releases are
> "3.3.6" and "3.4.4".
>
> There seem to be a few interesting bug fixes in the release notes for 3.3.5
> & 3.3.6:
> http://zookeeper.apache.org/doc/r3.3.5/releasenotes.html#changes
> http://zookeeper.apache.org/doc/r3.3.6/releasenotes.html#changes
>
> Notably fixes for:
>   leaking file handles
>   data inconsistencies and unexpired ephemeral nodes after cluster restart
>   shutdown synchronization
>
> But, I have not been using kafka long enough to see these issues myself.
>
>
>
> On Tue, Nov 13, 2012 at 3:29 PM, Neha Narkhede <[EMAIL PROTECTED]>wrote:
>
>> > So, there are known api changes that would preclude use of 3.4.x with
>> > kafka, it sounds like?
>>
>> Not really. Zookeeper 3.4.x is backwards compatible. Kafka 0.8 should
>> still work with any of the stable zookeeper versions out there. Once
>> ZK 3.4.x is stable, Kafka 0.8 would certainly benefit from the multi
>> APIs, since instead of doing one write per partition, we can do one
>> write for all partitions.
>>
>> > Is there a notion of a supported version of zookeeper for each kafka
>> > release?
>>
>> Not formally, though this sounds like a good idea. Implicitly, we
>> consider the zk version we release with as stable.
>>
>> > How about upgrading the recommended version for 0.8 at least to the
>> latest
>> > 3.3.x (3.3.6)?
>>
>> Yes, do you have an idea about any particular bugs that were fixed
>> post 3.3.4 which could be useful ?
>>
>> Thanks,
>> Neha
>>
>> On Tue, Nov 13, 2012 at 9:54 AM, Jason Rosenberg <[EMAIL PROTECTED]> wrote:
>> > Ok,
>> >
>> > So, there are known api changes that would preclude use of 3.4.x with
>> > kafka, it sounds like?
>> >
>> > Is there a notion of a supported version of zookeeper for each kafka
>> > release?
>> >
>> > How about upgrading the recommended version for 0.8 at least to the
>> latest
>> > 3.3.x (3.3.6)?
>> >
>> > Jason
>> >
>> >
>> >
>> > On Tue, Nov 13, 2012 at 8:50 AM, Neha Narkhede <[EMAIL PROTECTED]
>> >wrote:
>> >
>> >> I've been observing the zookeeper mailing list for 3.4.x and haven't
>> >> found it to be particularly stable. There are still serious issues
>> >> being reported like leader election failures, ephemeral node deletion.
>> >> I'm sure they are being fixed and there are many early adopters. But
>> >> my guess is that we should probably release 0.8 on a proven stable
>> >> version and look into 3.4.x for Kafka 0.8.1.
>> >>
>> >> Thanks,
>> >> Neha
>> >>
>> >> On Tue, Nov 13, 2012 at 6:57 AM, Jun Rao <[EMAIL PROTECTED]> wrote:
>> >> > Jason,
>> >> >
>> >> > We are just not sure about the stability of ZK 3.4.4. Does anybody
>> know?
>> >> In
>> >> > fact, some of the features in ZK 3.4.x (like the multi operation) can
>> >> > improve the availability of Kafka in 0.8. So, we'd like to move to ZK
>> >> 3.4.4
>> >> > if it's been widely used and is stable.
>> >> >
>> >> > Thanks,
>> >> >
>> >> > Jun
>> >> >
>> >> > On Mon, Nov 12, 2012 at 11:30 PM, Jason Rosenberg <[EMAIL PROTECTED]>
>> >> wrote:
>> >> >
>> >> >> Hi,
>> >> >>
>> >> >> I'm wondering if there's a reason kafka currently seems to depend on
>> >> >> zookeeper 3.3.4.  While the current stable version for zookeeper is
>> >> 3.4.4.
>> >> >>
>> >> >> Are there known issues with upgrading to the newer version of
>> zookeeper?
>> >> >>
>> >> >> Jason
>> >> >>
>> >>
>>

On Tue, Nov 13, 2012 at 12:30 AM, Jason Rosenberg <[EMAIL PROTECTED]> wrote:

> I'm wondering if there's a reason kafka currently seems to depend on
> zookeeper 3.3.4.
>

Looking at this<https://github.com/linkedin/linkedin-zookeeper/blob/master/project-spec.groovy>
3.3.3
is the currently HEAD in the repo. There is a pending pull
request<https://github.com/linkedin/linkedin-zookeeper/pull/9> for
3.3.4. Is that what your referencing?

Joe

If you do ./sbt make-pom, it generates a pom with the version set to 3.3.4.
 And I do think that is the current version (if you look at the release
notes, I think for 0.7.1, it mentions that one of the changes is to upgrade
to 3.3.4 zookeeper client).

Jason

On Tue, Nov 13, 2012 at 12:38 PM, Joseph Crotty <[EMAIL PROTECTED]>wrote:

> On Tue, Nov 13, 2012 at 12:30 AM, Jason Rosenberg <[EMAIL PROTECTED]>
> wrote:
>
> > I'm wondering if there's a reason kafka currently seems to depend on
> > zookeeper 3.3.4.
> >
>
> Looking at this<
> https://github.com/linkedin/linkedin-zookeeper/blob/master/project-spec.groovy
> >
> 3.3.3
> is the currently HEAD in the repo. There is a pending pull
> request<https://github.com/linkedin/linkedin-zookeeper/pull/9> for
> 3.3.4. Is that what your referencing?
>
> Joe
>