Hi guys,

I am running a kafka instance and a zookeeper instance on the same ec2
instance. I have 2 consumer groups on the same topic. They work fine for
few days but one of the consumers keeps stopping after few days. I can't
see any exception in my application logs or kafka log. The thread of the
stopping consumer is still running but consumer can't receive any message.
Even I restart my application and it's still the same. I have to restart
zookeeper and kafka to make it back to normal.

I used the following tool suggested by kafka docs and found owner is null
after consumer stops working. also its consumer offset won't change any
more.

bin/kafka-run-class.sh kafka.tools.ConsumerOffsetChecker --group mygroup
--zkconnect 127.0.0.1:2181 --topic mytopic

It's very similar with
http://mail-archives.apache.org/mod_mbox/incubator-kafka-users/201207.mbox/%[EMAIL PROTECTED]%3E

Any ideas about it?

Thanks

Ke

We have experienced the same issue and are still trying to figure out the
cause.
On Aug 25, 2012 1:29 PM, "Ke Ren" <[EMAIL PROTECTED]> wrote:

> Hi guys,
>
> I am running a kafka instance and a zookeeper instance on the same ec2
> instance. I have 2 consumer groups on the same topic. They work fine for
> few days but one of the consumers keeps stopping after few days. I can't
> see any exception in my application logs or kafka log. The thread of the
> stopping consumer is still running but consumer can't receive any message.
> Even I restart my application and it's still the same. I have to restart
> zookeeper and kafka to make it back to normal.
>
> I used the following tool suggested by kafka docs and found owner is null
> after consumer stops working. also its consumer offset won't change any
> more.
>
> bin/kafka-run-class.sh kafka.tools.ConsumerOffsetChecker --group mygroup
> --zkconnect 127.0.0.1:2181 --topic mytopic
>
> It's very similar with
>
> http://mail-archives.apache.org/mod_mbox/incubator-kafka-users/201207.mbox/%[EMAIL PROTECTED]%3E
>
> Any ideas about it?
>
> Thanks
>
> Ke
>

Hi

Since I am using a different consumer-group from the one that caused me
these problems, I haven't encountered these problems again.....

On 08/25/2012 11:57 PM, Patricio Echag�e wrote:
> We have experienced the same issue and are still trying to figure out the
> cause.
> On Aug 25, 2012 1:29 PM, "Ke Ren" <[EMAIL PROTECTED]> wrote:
>
>> Hi guys,
>>
>> I am running a kafka instance and a zookeeper instance on the same ec2
>> instance. I have 2 consumer groups on the same topic. They work fine for
>> few days but one of the consumers keeps stopping after few days. I can't
>> see any exception in my application logs or kafka log. The thread of the
>> stopping consumer is still running but consumer can't receive any message.
>> Even I restart my application and it's still the same. I have to restart
>> zookeeper and kafka to make it back to normal.
>>
>> I used the following tool suggested by kafka docs and found owner is null
>> after consumer stops working. also its consumer offset won't change any
>> more.
>>
>> bin/kafka-run-class.sh kafka.tools.ConsumerOffsetChecker --group mygroup
>> --zkconnect 127.0.0.1:2181 --topic mytopic
>>
>> It's very similar with
>>
>> http://mail-archives.apache.org/mod_mbox/incubator-kafka-users/201207.mbox/%[EMAIL PROTECTED]%3E
>>
>> Any ideas about it?
>>
>> Thanks
>>
>> Ke
>>

Hi Guy,

which kafka and zookeeper version are you using? what does your group id
look like? are you using number or string characters? We are using kafka
0.7.1 with the zookeeper in its package. I am going to try different group
id. The strange thing is we are using group name like xxxservice and it's
always the same consumer stopping. another consumer always works fine.

Thanks,

Ke

On Sun, Aug 26, 2012 at 6:54 AM, Guy Doulberg <[EMAIL PROTECTED]>wrote:

> Hi
>
> Since I am using a different consumer-group from the one that caused me
> these problems, I haven't encountered these problems again.....
>
>
>
>
> On 08/25/2012 11:57 PM, Patricio Echagüe wrote:
>
>> We have experienced the same issue and are still trying to figure out the
>> cause.
>> On Aug 25, 2012 1:29 PM, "Ke Ren" <[EMAIL PROTECTED]> wrote:
>>
>>  Hi guys,
>>>
>>> I am running a kafka instance and a zookeeper instance on the same ec2
>>> instance. I have 2 consumer groups on the same topic. They work fine for
>>> few days but one of the consumers keeps stopping after few days. I can't
>>> see any exception in my application logs or kafka log. The thread of the
>>> stopping consumer is still running but consumer can't receive any
>>> message.
>>> Even I restart my application and it's still the same. I have to restart
>>> zookeeper and kafka to make it back to normal.
>>>
>>> I used the following tool suggested by kafka docs and found owner is null
>>> after consumer stops working. also its consumer offset won't change any
>>> more.
>>>
>>> bin/kafka-run-class.sh kafka.tools.**ConsumerOffsetChecker --group
>>> mygroup
>>> --zkconnect 127.0.0.1:2181 --topic mytopic
>>>
>>> It's very similar with
>>>
>>> http://mail-archives.apache.**org/mod_mbox/incubator-kafka-**
>>> users/201207.mbox/%3C4FF59B1A.**[EMAIL PROTECTED]%3E<http://mail-archives.apache.org/mod_mbox/incubator-kafka-users/201207.mbox/%[EMAIL PROTECTED]%3E>
>>>
>>> Any ideas about it?
>>>
>>> Thanks
>>>
>>> Ke
>>>
>>>
>
>

We are using 0.7.1 and the zookeeper released with this version
and a name with alphanumberic characters...

In my case each time is was a different consumer stopping

Hope that helps

On 08/26/2012 11:07 AM, Ke Ren wrote:
> Hi Guy,
>
> which kafka and zookeeper version are you using? what does your group id
> look like? are you using number or string characters? We are using kafka
> 0.7.1 with the zookeeper in its package. I am going to try different group
> id. The strange thing is we are using group name like xxxservice and it's
> always the same consumer stopping. another consumer always works fine.
>
> Thanks,
>
> Ke
>
> On Sun, Aug 26, 2012 at 6:54 AM, Guy Doulberg <[EMAIL PROTECTED]>wrote:
>
>> Hi
>>
>> Since I am using a different consumer-group from the one that caused me
>> these problems, I haven't encountered these problems again.....
>>
>>
>>
>>
>> On 08/25/2012 11:57 PM, Patricio Echag�e wrote:
>>
>>> We have experienced the same issue and are still trying to figure out the
>>> cause.
>>> On Aug 25, 2012 1:29 PM, "Ke Ren" <[EMAIL PROTECTED]> wrote:
>>>
>>>   Hi guys,
>>>> I am running a kafka instance and a zookeeper instance on the same ec2
>>>> instance. I have 2 consumer groups on the same topic. They work fine for
>>>> few days but one of the consumers keeps stopping after few days. I can't
>>>> see any exception in my application logs or kafka log. The thread of the
>>>> stopping consumer is still running but consumer can't receive any
>>>> message.
>>>> Even I restart my application and it's still the same. I have to restart
>>>> zookeeper and kafka to make it back to normal.
>>>>
>>>> I used the following tool suggested by kafka docs and found owner is null
>>>> after consumer stops working. also its consumer offset won't change any
>>>> more.
>>>>
>>>> bin/kafka-run-class.sh kafka.tools.**ConsumerOffsetChecker --group
>>>> mygroup
>>>> --zkconnect 127.0.0.1:2181 --topic mytopic
>>>>
>>>> It's very similar with
>>>>
>>>> http://mail-archives.apache.**org/mod_mbox/incubator-kafka-**
>>>> users/201207.mbox/%3C4FF59B1A.**[EMAIL PROTECTED]%3E<http://mail-archives.apache.org/mod_mbox/incubator-kafka-users/201207.mbox/%[EMAIL PROTECTED]%3E>
>>>>
>>>> Any ideas about it?
>>>>
>>>> Thanks
>>>>
>>>> Ke
>>>>
>>>>
>>

Ke,

Have your tried suggestions in item #3 in
http://incubator.apache.org/kafka/faq.html ? If you put your consumer
thread in a java threadpool, exceptions may be eaten.

Thanks,

Jun

On Sat, Aug 25, 2012 at 1:29 PM, Ke Ren <[EMAIL PROTECTED]> wrote:

> Hi guys,
>
> I am running a kafka instance and a zookeeper instance on the same ec2
> instance. I have 2 consumer groups on the same topic. They work fine for
> few days but one of the consumers keeps stopping after few days. I can't
> see any exception in my application logs or kafka log. The thread of the
> stopping consumer is still running but consumer can't receive any message.
> Even I restart my application and it's still the same. I have to restart
> zookeeper and kafka to make it back to normal.
>
> I used the following tool suggested by kafka docs and found owner is null
> after consumer stops working. also its consumer offset won't change any
> more.
>
> bin/kafka-run-class.sh kafka.tools.ConsumerOffsetChecker --group mygroup
> --zkconnect 127.0.0.1:2181 --topic mytopic
>
> It's very similar with
>
> http://mail-archives.apache.org/mod_mbox/incubator-kafka-users/201207.mbox/%[EMAIL PROTECTED]%3E
>
> Any ideas about it?
>
> Thanks
>
> Ke
>

Hi Jun,

yes, we tried that. we also added mbean over jmx to monitor. The thread is
healthy. Moreover, if it's something wrong with application, it should
recover after restarting. However, after restarting application, it
received few messages and immediately died. We have to restart kafka and
zookeeper to make it back to normal. I haven't identified whether those few
messages received after restarting are new messages from topic or always
few repeated messages.

do you know what "Owner = null" means and why it happens? If the consumer
works properly, owner always equals something like:

Owner = xxxservice_ip-10-60-41-167-1345969396925-4966bd1a-0

If there is no consumer on a group, ConsumerOffsetChecker won't return
anything. We always get Owner = null when consumer stops working. Any ideas?

Thanks,

Ke

On Sun, Aug 26, 2012 at 6:10 PM, Jun Rao <[EMAIL PROTECTED]> wrote:

> Ke,
>
> Have your tried suggestions in item #3 in
> http://incubator.apache.org/kafka/faq.html ? If you put your consumer
> thread in a java threadpool, exceptions may be eaten.
>
> Thanks,
>
> Jun
>
> On Sat, Aug 25, 2012 at 1:29 PM, Ke Ren <[EMAIL PROTECTED]> wrote:
>
> > Hi guys,
> >
> > I am running a kafka instance and a zookeeper instance on the same ec2
> > instance. I have 2 consumer groups on the same topic. They work fine for
> > few days but one of the consumers keeps stopping after few days. I can't
> > see any exception in my application logs or kafka log. The thread of the
> > stopping consumer is still running but consumer can't receive any
> message.
> > Even I restart my application and it's still the same. I have to restart
> > zookeeper and kafka to make it back to normal.
> >
> > I used the following tool suggested by kafka docs and found owner is null
> > after consumer stops working. also its consumer offset won't change any
> > more.
> >
> > bin/kafka-run-class.sh kafka.tools.ConsumerOffsetChecker --group mygroup
> > --zkconnect 127.0.0.1:2181 --topic mytopic
> >
> > It's very similar with
> >
> >
> http://mail-archives.apache.org/mod_mbox/incubator-kafka-users/201207.mbox/%[EMAIL PROTECTED]%3E
> >
> > Any ideas about it?
> >
> > Thanks
> >
> > Ke
> >
>

In my case I'm using Kafka 0.7.1 , zookeeper 3.3.5 and storm 0.8

I also saw the owner = null when querying zookeeper.

I can't be more specific though on when it happens. I blame it on storm
worker restarting multiple times when it faces other errors in different
threads within the same worker.
On Aug 26, 2012 10:28 AM, "Ke Ren" <[EMAIL PROTECTED]> wrote:

> Hi Jun,
>
> yes, we tried that. we also added mbean over jmx to monitor. The thread is
> healthy. Moreover, if it's something wrong with application, it should
> recover after restarting. However, after restarting application, it
> received few messages and immediately died. We have to restart kafka and
> zookeeper to make it back to normal. I haven't identified whether those few
> messages received after restarting are new messages from topic or always
> few repeated messages.
>
> do you know what "Owner = null" means and why it happens? If the consumer
> works properly, owner always equals something like:
>
> Owner = xxxservice_ip-10-60-41-167-1345969396925-4966bd1a-0
>
> If there is no consumer on a group, ConsumerOffsetChecker won't return
> anything. We always get Owner = null when consumer stops working. Any
> ideas?
>
> Thanks,
>
> Ke
>
> On Sun, Aug 26, 2012 at 6:10 PM, Jun Rao <[EMAIL PROTECTED]> wrote:
>
> > Ke,
> >
> > Have your tried suggestions in item #3 in
> > http://incubator.apache.org/kafka/faq.html ? If you put your consumer
> > thread in a java threadpool, exceptions may be eaten.
> >
> > Thanks,
> >
> > Jun
> >
> > On Sat, Aug 25, 2012 at 1:29 PM, Ke Ren <[EMAIL PROTECTED]> wrote:
> >
> > > Hi guys,
> > >
> > > I am running a kafka instance and a zookeeper instance on the same ec2
> > > instance. I have 2 consumer groups on the same topic. They work fine
> for
> > > few days but one of the consumers keeps stopping after few days. I
> can't
> > > see any exception in my application logs or kafka log. The thread of
> the
> > > stopping consumer is still running but consumer can't receive any
> > message.
> > > Even I restart my application and it's still the same. I have to
> restart
> > > zookeeper and kafka to make it back to normal.
> > >
> > > I used the following tool suggested by kafka docs and found owner is
> null
> > > after consumer stops working. also its consumer offset won't change any
> > > more.
> > >
> > > bin/kafka-run-class.sh kafka.tools.ConsumerOffsetChecker --group
> mygroup
> > > --zkconnect 127.0.0.1:2181 --topic mytopic
> > >
> > > It's very similar with
> > >
> > >
> >
> http://mail-archives.apache.org/mod_mbox/incubator-kafka-users/201207.mbox/%[EMAIL PROTECTED]%3E
> > >
> > > Any ideas about it?
> > >
> > > Thanks
> > >
> > > Ke
> > >
> >
>

"Owner = null" means that a partition is not consumed by any consumer.
Typically, this only happens transiently when consumes rebalance among
themselves. Do you see lots of rebalances in your consumer log? Do you see
any ZK session expiration?

Thanks,

Jun

On Sun, Aug 26, 2012 at 10:27 AM, Ke Ren <[EMAIL PROTECTED]> wrote:

> Hi Jun,
>
> yes, we tried that. we also added mbean over jmx to monitor. The thread is
> healthy. Moreover, if it's something wrong with application, it should
> recover after restarting. However, after restarting application, it
> received few messages and immediately died. We have to restart kafka and
> zookeeper to make it back to normal. I haven't identified whether those few
> messages received after restarting are new messages from topic or always
> few repeated messages.
>
> do you know what "Owner = null" means and why it happens? If the consumer
> works properly, owner always equals something like:
>
> Owner = xxxservice_ip-10-60-41-167-1345969396925-4966bd1a-0
>
> If there is no consumer on a group, ConsumerOffsetChecker won't return
> anything. We always get Owner = null when consumer stops working. Any
> ideas?
>
> Thanks,
>
> Ke
>
> On Sun, Aug 26, 2012 at 6:10 PM, Jun Rao <[EMAIL PROTECTED]> wrote:
>
> > Ke,
> >
> > Have your tried suggestions in item #3 in
> > http://incubator.apache.org/kafka/faq.html ? If you put your consumer
> > thread in a java threadpool, exceptions may be eaten.
> >
> > Thanks,
> >
> > Jun
> >
> > On Sat, Aug 25, 2012 at 1:29 PM, Ke Ren <[EMAIL PROTECTED]> wrote:
> >
> > > Hi guys,
> > >
> > > I am running a kafka instance and a zookeeper instance on the same ec2
> > > instance. I have 2 consumer groups on the same topic. They work fine
> for
> > > few days but one of the consumers keeps stopping after few days. I
> can't
> > > see any exception in my application logs or kafka log. The thread of
> the
> > > stopping consumer is still running but consumer can't receive any
> > message.
> > > Even I restart my application and it's still the same. I have to
> restart
> > > zookeeper and kafka to make it back to normal.
> > >
> > > I used the following tool suggested by kafka docs and found owner is
> null
> > > after consumer stops working. also its consumer offset won't change any
> > > more.
> > >
> > > bin/kafka-run-class.sh kafka.tools.ConsumerOffsetChecker --group
> mygroup
> > > --zkconnect 127.0.0.1:2181 --topic mytopic
> > >
> > > It's very similar with
> > >
> > >
> >
> http://mail-archives.apache.org/mod_mbox/incubator-kafka-users/201207.mbox/%[EMAIL PROTECTED]%3E
> > >
> > > Any ideas about it?
> > >
> > > Thanks
> > >
> > > Ke
> > >
> >
>

Cool, thanks for your explanation. We are quite new to kafka and zookeeper.
just adopt kafka as a part of our project recently.

It sounds like something failed during rebalancing. I don't have access to
our log atm. Will let you know once I do. Thanks for pointing a right
direction for us.

Cheers,

Ke

On Mon, Aug 27, 2012 at 12:40 AM, Jun Rao <[EMAIL PROTECTED]> wrote:

>  "Owner = null" means that a partition is not consumed by any consumer.
> Typically, this only happens transiently when consumes rebalance among
> themselves. Do you see lots of rebalances in your consumer log? Do you see
> any ZK session expiration?
>
> Thanks,
>
> Jun
>
> On Sun, Aug 26, 2012 at 10:27 AM, Ke Ren <[EMAIL PROTECTED]> wrote:
>
> > Hi Jun,
> >
> > yes, we tried that. we also added mbean over jmx to monitor. The thread
> is
> > healthy. Moreover, if it's something wrong with application, it should
> > recover after restarting. However, after restarting application, it
> > received few messages and immediately died. We have to restart kafka and
> > zookeeper to make it back to normal. I haven't identified whether those
> few
> > messages received after restarting are new messages from topic or always
> > few repeated messages.
> >
> > do you know what "Owner = null" means and why it happens? If the consumer
> > works properly, owner always equals something like:
> >
> > Owner = xxxservice_ip-10-60-41-167-1345969396925-4966bd1a-0
> >
> > If there is no consumer on a group, ConsumerOffsetChecker won't return
> > anything. We always get Owner = null when consumer stops working. Any
> > ideas?
> >
> > Thanks,
> >
> > Ke
> >
> > On Sun, Aug 26, 2012 at 6:10 PM, Jun Rao <[EMAIL PROTECTED]> wrote:
> >
> > > Ke,
> > >
> > > Have your tried suggestions in item #3 in
> > > http://incubator.apache.org/kafka/faq.html ? If you put your consumer
> > > thread in a java threadpool, exceptions may be eaten.
> > >
> > > Thanks,
> > >
> > > Jun
> > >
> > > On Sat, Aug 25, 2012 at 1:29 PM, Ke Ren <[EMAIL PROTECTED]> wrote:
> > >
> > > > Hi guys,
> > > >
> > > > I am running a kafka instance and a zookeeper instance on the same
> ec2
> > > > instance. I have 2 consumer groups on the same topic. They work fine
> > for
> > > > few days but one of the consumers keeps stopping after few days. I
> > can't
> > > > see any exception in my application logs or kafka log. The thread of
> > the
> > > > stopping consumer is still running but consumer can't receive any
> > > message.
> > > > Even I restart my application and it's still the same. I have to
> > restart
> > > > zookeeper and kafka to make it back to normal.
> > > >
> > > > I used the following tool suggested by kafka docs and found owner is
> > null
> > > > after consumer stops working. also its consumer offset won't change
> any
> > > > more.
> > > >
> > > > bin/kafka-run-class.sh kafka.tools.ConsumerOffsetChecker --group
> > mygroup
> > > > --zkconnect 127.0.0.1:2181 --topic mytopic
> > > >
> > > > It's very similar with
> > > >
> > > >
> > >
> >
> http://mail-archives.apache.org/mod_mbox/incubator-kafka-users/201207.mbox/%[EMAIL PROTECTED]%3E
> > > >
> > > > Any ideas about it?
> > > >
> > > > Thanks
> > > >
> > > > Ke
> > > >
> > >
> >
>