Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # user >> Kafka producer - jump start


Copy link to this message
-
Re: Kafka producer - jump start
I would also like to add,

Another producer that shares the same code as the producer I restarted,
was running and didn't suffer any problems,

So, I am pretty sure there was a problem with the init process of the
producer, I will update you guys if ever happen again,
I hope this time I will save the jstack so we will  a reference point

On 10/03/2012 06:36 PM, Guy Doulberg wrote:
> Sorry.
>
> Unfourtently I didn't keep them...
>
> I realized it only after I solved the problem.....which was too late....
> I can't reproduce....
>
>
>
> Jun Rao <[EMAIL PROTECTED]> ן¿½ן¿½ן¿½:
>
>
> Guy,
>
> A single producer is fine. Under the cover, it will create 1 async producer
> thread per broker. Haven't seen this issue before. Could you post the stack
> trace?
>
> Thanks,
>
> Jun
>
> On Wed, Oct 3, 2012 at 8:27 AM, Guy Doulberg <[EMAIL PROTECTED]>wrote:
>
>> Hi guys,
>>
>> I had today a production problem with the Kafka producer (version 0.7.1),
>>
>> I have web front end HTTP server that produces events using the async
>> kafka producer.
>> The design is that the kafka producer is a singleton,
>>
>> I often restart that HTTP server, and most of the time I could see booting
>> up the Kafka producer doesn't take too long, and eventually I am ending up
>> with a single kafka producer...
>>
>>
>> Today something went wrong, and I could idnetify locks on my server, using
>> jstack I saw that all my locks are from the producer initializing,
>> specifically in the method:  getZKBrokerInfo
>>
>>
>> Thinking my bottleneck was the Zookeeper I restarted (one after another)
>> its nodes, this didn't solve the problem,
>>
>>
>> Without having a plan, I restarted one of the kafka borker (I have 3) ,
>> magically it solved the problem....
>>
>> I am trying to understand what was the problem,
>>
>> could it be that when  producer wakes up - it tries to connect each of the
>> brokers, and the broker I picked was not responding?  - I have service that
>> monitor load on my broker machines and I had no indication of such load,
>> moreover I have service that check if a broker responds to 9092 and this
>> broker responded.
>>
>> If a broker doesn't respond, shouldn't the producer drop  this broker, and
>> use only the others?
>>
>> Anyhow, I am not expecting any of you to know what exactly happened to my
>> producer, but I am trying to understand more the producer init process,
>> maybe I should create a connection to the kafka broker earlier in my app
>> life cycle.
>>
>> I am also trying to figure out why there is a synchronized code in the
>> producer, should I not use a single producer?
>>
>>
>> Thanks
>> Guy Doulberg
>> Data infrastructure engineer
>>
>>
>>
>>
>>
>>
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB