Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # user >> Getting timeouts with elastic load balancer in AWS


Copy link to this message
-
Re: Getting timeouts with elastic load balancer in AWS
Just to clarify: num.retries > 0 does not guarantee that all messages will
be received at the broker. It guarantees retry on exceptions - so it cannot
handle the corner case when the broker goes down after the message is
written to the socket buffer but before the buffer is flushed (in which
case no exceptions are thrown). This is addressed in 0.8 with producer acks.

That said, you have a fairly large interval between messages so it's rather
surprising. It might help to correlate this with broker-side logs to see if
the "Message sent" for message 5 was actually received on the broker.

Thanks,

Joel

On Thu, Jun 28, 2012 at 1:36 PM, Vaibhav Puranik <[EMAIL PROTECTED]> wrote:

> Jun,
>
> Here is the log with SynProducer and DefaultEventHandler trace enabled.
>
> http://pastebin.com/dTm5RSJ9
>
> Here are my producer settings:
>
> properties.put("serializer.class", "kafka.serializer.StringEncoder")
> properties.put("broker.list", "0:localhost:9092")
> properties.put("producer.type", "async");
> properties.put("num.retries", "3");
> properties.put("batch.size", "5");
>
> (This batch size does't work because I think the some flush time  is small
> - 5 seconds - It sends every message as it comes). I am sleeping for 15
> seconds between each messages.
>
> Here is my broker output:
> _____0_____  {�D�_____1_____  �&6c_____2_____  6z��_____3_____  +
> �~_____4_____  f�tu_____6_____  ����_____7_____  \� _____8_____
>  ��Ơ_____9_____
>
>
> Notice number 5 is missing. I restarted broker between 4 and 5. You can see
> that the message  5 is missing. On producer for some reason the error
> appears between 6 and 7. Don't know why.
>
> Regards,
> Vaibhav
>
>
> On Thu, Jun 28, 2012 at 11:15 AM, Jun Rao <[EMAIL PROTECTED]> wrote:
>
> > Could you enable trace logging in DefaultEventHandler to see if the
> > following message shows up after the warning?
> >          trace("kafka producer sent messages for topics %s to broker
> %s:%d
> > (on attempt %d)"
> >
> > Thanks,
> >
> > Jun
> >
> > On Thu, Jun 28, 2012 at 10:44 AM, Vaibhav Puranik <[EMAIL PROTECTED]
> > >wrote:
> >
> > > Hi all,
> > >
> > > I don't think the num.retries (0.7.1) is working. Here is how I tested
> > it.
> > >
> > > I wrote a simple producer that sends messages with the following
> strings
> > -
> > > "____1_____", "_____2_____"..... . As you can see all the messages are
> > > sequential.
> > > I tailed the topic log on broker. After sending every message, I have
> > added
> > > Thread.sleep for 15 seconds.
> > >
> > > Everytime I send the message, it immediately appears in the broker log.
> > But
> > > if I restart the broker to simulate producer connection drop (in the 15
> > > seconds producer sleep period), it prints the following message in the
> > > logs:
> > >
> > > [2012-06-28 10:31:17,127] INFO Disconnecting from localhost:9092
> > > (kafka.producer.SyncProducer)
> > > [2012-06-28 10:31:17,132] WARN Error sending messages, 2 attempts
> > remaining
> > > (kafka.producer.async.DefaultEventHandler)
> > > [2012-06-28 10:31:17,132] INFO Connected to localhost:9092 for
> producing
> > > (kafka.producer.SyncProducer)
> > >
> > > But the message that was sent right after the broker restart never
> > reaches
> > > the broker. The message after that (2nd message after restart) gets to
> > > broker fine and the sequence continues. Thus if I restart the broker in
> > the
> > > sleep period between message 4 and 5. I don't get the message 5. I get
> > > message 1,2,3,4,6,7,.....
> > >
> > > I tried setting num.retries to 1 and 2 thinking that in the first retry
> > it
> > > might reconnect and the second retry is where it's resending the
> message.
> > > But that doesn't work. Number of retries doesn't improve the situation.
> > >
> > > Can you see any flaw in my testing? What can I do to better test this
> > > scenario? How can I ensure that no messages are dropped? I don't think
> I
> > am
> > > loosing the message because it's in broker memory. Please correct me
> if I
> >
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB