Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Kafka >> mail # user >> Kafka producer - jump start


Copy link to this message
-
Kafka producer - jump start
Hi guys,

I had today a production problem with the Kafka producer (version 0.7.1),

I have web front end HTTP server that produces events using the async
kafka producer.
The design is that the kafka producer is a singleton,

I often restart that HTTP server, and most of the time I could see
booting up the Kafka producer doesn't take too long, and eventually I am
ending up with a single kafka producer...
Today something went wrong, and I could idnetify locks on my server,
using jstack I saw that all my locks are from the producer initializing,
specifically in the method:  getZKBrokerInfo
Thinking my bottleneck was the Zookeeper I restarted (one after another)
its nodes, this didn't solve the problem,
Without having a plan, I restarted one of the kafka borker (I have 3) ,
magically it solved the problem....

I am trying to understand what was the problem,

could it be that when  producer wakes up - it tries to connect each of
the brokers, and the broker I picked was not responding?  - I have
service that monitor load on my broker machines and I had no indication
of such load, moreover I have service that check if a broker responds to
9092 and this broker responded.

If a broker doesn't respond, shouldn't the producer drop  this broker,
and use only the others?

Anyhow, I am not expecting any of you to know what exactly happened to
my producer, but I am trying to understand more the producer init
process, maybe I should create a connection to the kafka broker earlier
in my app life cycle.

I am also trying to figure out why there is a synchronized code in the
producer, should I not use a single producer?
Thanks
Guy Doulberg
Data infrastructure engineer
+
Jun Rao 2012-10-03, 16:18
+
Guy Doulberg 2012-10-03, 16:36
+
Guy Doulberg 2012-10-04, 07:14
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB