Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # user >> Getting timeouts with elastic load balancer in AWS


Copy link to this message
-
Re: Getting timeouts with elastic load balancer in AWS
Hi all,

I don't think the num.retries (0.7.1) is working. Here is how I tested it.

I wrote a simple producer that sends messages with the following strings -
"____1_____", "_____2_____"..... . As you can see all the messages are
sequential.
I tailed the topic log on broker. After sending every message, I have added
Thread.sleep for 15 seconds.

Everytime I send the message, it immediately appears in the broker log. But
if I restart the broker to simulate producer connection drop (in the 15
seconds producer sleep period), it prints the following message in the logs:

[2012-06-28 10:31:17,127] INFO Disconnecting from localhost:9092
(kafka.producer.SyncProducer)
[2012-06-28 10:31:17,132] WARN Error sending messages, 2 attempts remaining
(kafka.producer.async.DefaultEventHandler)
[2012-06-28 10:31:17,132] INFO Connected to localhost:9092 for producing
(kafka.producer.SyncProducer)

But the message that was sent right after the broker restart never reaches
the broker. The message after that (2nd message after restart) gets to
broker fine and the sequence continues. Thus if I restart the broker in the
sleep period between message 4 and 5. I don't get the message 5. I get
message 1,2,3,4,6,7,.....

I tried setting num.retries to 1 and 2 thinking that in the first retry it
might reconnect and the second retry is where it's resending the message.
But that doesn't work. Number of retries doesn't improve the situation.

Can you see any flaw in my testing? What can I do to better test this
scenario? How can I ensure that no messages are dropped? I don't think I am
loosing the message because it's in broker memory. Please correct me if I
am wrong.

Regards,
Vaibhav
GumGum <http://gumgum.com>

On Wed, Jun 27, 2012 at 3:42 PM, Joel Koshy <[EMAIL PROTECTED]> wrote:

> 0.7.1 has this: reconnect.time.interval.ms
>
> Thanks,
>
> Joel
>
> On Wed, Jun 27, 2012 at 3:31 PM, Vaibhav Puranik <[EMAIL PROTECTED]>
> wrote:
>
> > That will be awesome. It will definitely address AWS ELB problem.
> >
> > +1 for "reconnect.interval".
> >
> > Regards,
> > Vaibhav
> > GumGum
> >
> >
> > On Wed, Jun 27, 2012 at 3:24 PM, Niek Sanders <[EMAIL PROTECTED]
> > >wrote:
> >
> > > Do producers currently leave the sockets to the brokers open
> > indefinitely?
> > >
> > > It might make sense to add a second producer config param similar to
> > > "reconnect.interval" which limits on time instead of message count.
> > > (And then reconnect based on whichever criteria is hit first).  For
> > > folks going through ELBs on AWS, they'd set the reconnect.interval.sec
> > > to something like 50 sec as a workaround for low-volume producers.
> > >
> > > - Niek
> > >
> > >
> > >
> > > On Tue, Jun 26, 2012 at 4:52 PM, Jun Rao <[EMAIL PROTECTED]> wrote:
> > > > Set num.retries in producer config property file. It defaults to 0.
> > > >
> > > > Thanks,
> > > >
> > > > Jun
> > > >
> > >
> >
>
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB