Hi guys,

I had today a production problem with the Kafka producer (version 0.7.1),

I have web front end HTTP server that produces events using the async
kafka producer.
The design is that the kafka producer is a singleton,

I often restart that HTTP server, and most of the time I could see
booting up the Kafka producer doesn't take too long, and eventually I am
ending up with a single kafka producer...
Today something went wrong, and I could idnetify locks on my server,
using jstack I saw that all my locks are from the producer initializing,
specifically in the method:  getZKBrokerInfo
Thinking my bottleneck was the Zookeeper I restarted (one after another)
its nodes, this didn't solve the problem,
Without having a plan, I restarted one of the kafka borker (I have 3) ,
magically it solved the problem....

I am trying to understand what was the problem,

could it be that when  producer wakes up - it tries to connect each of
the brokers, and the broker I picked was not responding?  - I have
service that monitor load on my broker machines and I had no indication
of such load, moreover I have service that check if a broker responds to
9092 and this broker responded.

If a broker doesn't respond, shouldn't the producer drop  this broker,
and use only the others?

Anyhow, I am not expecting any of you to know what exactly happened to
my producer, but I am trying to understand more the producer init
process, maybe I should create a connection to the kafka broker earlier
in my app life cycle.

I am also trying to figure out why there is a synchronized code in the
producer, should I not use a single producer?
Thanks
Guy Doulberg
Data infrastructure engineer

Guy,

A single producer is fine. Under the cover, it will create 1 async producer
thread per broker. Haven't seen this issue before. Could you post the stack
trace?

Thanks,

Jun

On Wed, Oct 3, 2012 at 8:27 AM, Guy Doulberg <[EMAIL PROTECTED]>wrote:

> Hi guys,
>
> I had today a production problem with the Kafka producer (version 0.7.1),
>
> I have web front end HTTP server that produces events using the async
> kafka producer.
> The design is that the kafka producer is a singleton,
>
> I often restart that HTTP server, and most of the time I could see booting
> up the Kafka producer doesn't take too long, and eventually I am ending up
> with a single kafka producer...
>
>
> Today something went wrong, and I could idnetify locks on my server, using
> jstack I saw that all my locks are from the producer initializing,
> specifically in the method:  getZKBrokerInfo
>
>
> Thinking my bottleneck was the Zookeeper I restarted (one after another)
> its nodes, this didn't solve the problem,
>
>
> Without having a plan, I restarted one of the kafka borker (I have 3) ,
> magically it solved the problem....
>
> I am trying to understand what was the problem,
>
> could it be that when  producer wakes up - it tries to connect each of the
> brokers, and the broker I picked was not responding?  - I have service that
> monitor load on my broker machines and I had no indication of such load,
> moreover I have service that check if a broker responds to 9092 and this
> broker responded.
>
> If a broker doesn't respond, shouldn't the producer drop  this broker, and
> use only the others?
>
> Anyhow, I am not expecting any of you to know what exactly happened to my
> producer, but I am trying to understand more the producer init process,
> maybe I should create a connection to the kafka broker earlier in my app
> life cycle.
>
> I am also trying to figure out why there is a synchronized code in the
> producer, should I not use a single producer?
>
>
> Thanks
> Guy Doulberg
> Data infrastructure engineer
>
>
>
>
>
>

Sorry.

Unfourtently I didn't keep them...

I realized it only after I solved the problem.....which was too late....
I can't reproduce....

Jun Rao <[EMAIL PROTECTED]> כתב:
Guy,

A single producer is fine. Under the cover, it will create 1 async producer
thread per broker. Haven't seen this issue before. Could you post the stack
trace?

Thanks,

Jun

On Wed, Oct 3, 2012 at 8:27 AM, Guy Doulberg <[EMAIL PROTECTED]>wrote:

> Hi guys,
>
> I had today a production problem with the Kafka producer (version 0.7.1),
>
> I have web front end HTTP server that produces events using the async
> kafka producer.
> The design is that the kafka producer is a singleton,
>
> I often restart that HTTP server, and most of the time I could see booting
> up the Kafka producer doesn't take too long, and eventually I am ending up
> with a single kafka producer...
>
>
> Today something went wrong, and I could idnetify locks on my server, using
> jstack I saw that all my locks are from the producer initializing,
> specifically in the method:  getZKBrokerInfo
>
>
> Thinking my bottleneck was the Zookeeper I restarted (one after another)
> its nodes, this didn't solve the problem,
>
>
> Without having a plan, I restarted one of the kafka borker (I have 3) ,
> magically it solved the problem....
>
> I am trying to understand what was the problem,
>
> could it be that when  producer wakes up - it tries to connect each of the
> brokers, and the broker I picked was not responding?  - I have service that
> monitor load on my broker machines and I had no indication of such load,
> moreover I have service that check if a broker responds to 9092 and this
> broker responded.
>
> If a broker doesn't respond, shouldn't the producer drop  this broker, and
> use only the others?
>
> Anyhow, I am not expecting any of you to know what exactly happened to my
> producer, but I am trying to understand more the producer init process,
> maybe I should create a connection to the kafka broker earlier in my app
> life cycle.
>
> I am also trying to figure out why there is a synchronized code in the
> producer, should I not use a single producer?
>
>
> Thanks
> Guy Doulberg
> Data infrastructure engineer
>
>
>
>
>
>

I would also like to add,

Another producer that shares the same code as the producer I restarted,
was running and didn't suffer any problems,

So, I am pretty sure there was a problem with the init process of the
producer, I will update you guys if ever happen again,
I hope this time I will save the jstack so we will  a reference point

On 10/03/2012 06:36 PM, Guy Doulberg wrote:
> Sorry.
>
> Unfourtently I didn't keep them...
>
> I realized it only after I solved the problem.....which was too late....
> I can't reproduce....
>
>
>
> Jun Rao <[EMAIL PROTECTED]> ן¿½ן¿½ן¿½:
>
>
> Guy,
>
> A single producer is fine. Under the cover, it will create 1 async producer
> thread per broker. Haven't seen this issue before. Could you post the stack
> trace?
>
> Thanks,
>
> Jun
>
> On Wed, Oct 3, 2012 at 8:27 AM, Guy Doulberg <[EMAIL PROTECTED]>wrote:
>
>> Hi guys,
>>
>> I had today a production problem with the Kafka producer (version 0.7.1),
>>
>> I have web front end HTTP server that produces events using the async
>> kafka producer.
>> The design is that the kafka producer is a singleton,
>>
>> I often restart that HTTP server, and most of the time I could see booting
>> up the Kafka producer doesn't take too long, and eventually I am ending up
>> with a single kafka producer...
>>
>>
>> Today something went wrong, and I could idnetify locks on my server, using
>> jstack I saw that all my locks are from the producer initializing,
>> specifically in the method:  getZKBrokerInfo
>>
>>
>> Thinking my bottleneck was the Zookeeper I restarted (one after another)
>> its nodes, this didn't solve the problem,
>>
>>
>> Without having a plan, I restarted one of the kafka borker (I have 3) ,
>> magically it solved the problem....
>>
>> I am trying to understand what was the problem,
>>
>> could it be that when  producer wakes up - it tries to connect each of the
>> brokers, and the broker I picked was not responding?  - I have service that
>> monitor load on my broker machines and I had no indication of such load,
>> moreover I have service that check if a broker responds to 9092 and this
>> broker responded.
>>
>> If a broker doesn't respond, shouldn't the producer drop  this broker, and
>> use only the others?
>>
>> Anyhow, I am not expecting any of you to know what exactly happened to my
>> producer, but I am trying to understand more the producer init process,
>> maybe I should create a connection to the kafka broker earlier in my app
>> life cycle.
>>
>> I am also trying to figure out why there is a synchronized code in the
>> producer, should I not use a single producer?
>>
>>
>> Thanks
>> Guy Doulberg
>> Data infrastructure engineer
>>
>>
>>
>>
>>
>>