Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # user >> Getting timeouts with elastic load balancer in AWS


Copy link to this message
-
Re: Getting timeouts with elastic load balancer in AWS
Jun,

Here is the log with SynProducer and DefaultEventHandler trace enabled.

http://pastebin.com/dTm5RSJ9

Here are my producer settings:

properties.put("serializer.class", "kafka.serializer.StringEncoder")
properties.put("broker.list", "0:localhost:9092")
properties.put("producer.type", "async");
properties.put("num.retries", "3");
properties.put("batch.size", "5");

(This batch size does't work because I think the some flush time  is small
- 5 seconds - It sends every message as it comes). I am sleeping for 15
seconds between each messages.

Here is my broker output:
_____0_____#17;#1;{�D�_____1_____#17;#1;�&6c_____2_____#17;#1;6z��_____3_____#17;#1;+�~_____4_____#17;#1;f�tu_____6_____#17;#1;����_____7_____#17;#1;\�#21;_____8_____#17;#1;��Ơ_____9_____
Notice number 5 is missing. I restarted broker between 4 and 5. You can see
that the message  5 is missing. On producer for some reason the error
appears between 6 and 7. Don't know why.

Regards,
Vaibhav
On Thu, Jun 28, 2012 at 11:15 AM, Jun Rao <[EMAIL PROTECTED]> wrote:

> Could you enable trace logging in DefaultEventHandler to see if the
> following message shows up after the warning?
>          trace("kafka producer sent messages for topics %s to broker %s:%d
> (on attempt %d)"
>
> Thanks,
>
> Jun
>
> On Thu, Jun 28, 2012 at 10:44 AM, Vaibhav Puranik <[EMAIL PROTECTED]
> >wrote:
>
> > Hi all,
> >
> > I don't think the num.retries (0.7.1) is working. Here is how I tested
> it.
> >
> > I wrote a simple producer that sends messages with the following strings
> -
> > "____1_____", "_____2_____"..... . As you can see all the messages are
> > sequential.
> > I tailed the topic log on broker. After sending every message, I have
> added
> > Thread.sleep for 15 seconds.
> >
> > Everytime I send the message, it immediately appears in the broker log.
> But
> > if I restart the broker to simulate producer connection drop (in the 15
> > seconds producer sleep period), it prints the following message in the
> > logs:
> >
> > [2012-06-28 10:31:17,127] INFO Disconnecting from localhost:9092
> > (kafka.producer.SyncProducer)
> > [2012-06-28 10:31:17,132] WARN Error sending messages, 2 attempts
> remaining
> > (kafka.producer.async.DefaultEventHandler)
> > [2012-06-28 10:31:17,132] INFO Connected to localhost:9092 for producing
> > (kafka.producer.SyncProducer)
> >
> > But the message that was sent right after the broker restart never
> reaches
> > the broker. The message after that (2nd message after restart) gets to
> > broker fine and the sequence continues. Thus if I restart the broker in
> the
> > sleep period between message 4 and 5. I don't get the message 5. I get
> > message 1,2,3,4,6,7,.....
> >
> > I tried setting num.retries to 1 and 2 thinking that in the first retry
> it
> > might reconnect and the second retry is where it's resending the message.
> > But that doesn't work. Number of retries doesn't improve the situation.
> >
> > Can you see any flaw in my testing? What can I do to better test this
> > scenario? How can I ensure that no messages are dropped? I don't think I
> am
> > loosing the message because it's in broker memory. Please correct me if I
> > am wrong.
> >
> > Regards,
> > Vaibhav
> > GumGum <http://gumgum.com>
> >
> >
> >
> > On Wed, Jun 27, 2012 at 3:42 PM, Joel Koshy <[EMAIL PROTECTED]> wrote:
> >
> > > 0.7.1 has this: reconnect.time.interval.ms
> > >
> > > Thanks,
> > >
> > > Joel
> > >
> > > On Wed, Jun 27, 2012 at 3:31 PM, Vaibhav Puranik <[EMAIL PROTECTED]>
> > > wrote:
> > >
> > > > That will be awesome. It will definitely address AWS ELB problem.
> > > >
> > > > +1 for "reconnect.interval".
> > > >
> > > > Regards,
> > > > Vaibhav
> > > > GumGum
> > > >
> > > >
> > > > On Wed, Jun 27, 2012 at 3:24 PM, Niek Sanders <
> [EMAIL PROTECTED]
> > > > >wrote:
> > > >
> > > > > Do producers currently leave the sockets to the brokers open
> > > > indefinitely?
> > > > >
> > > > > It might make sense to add a second producer config param similar