Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka, mail # user - Getting timeouts with elastic load balancer in AWS


Copy link to this message
-
Re: Getting timeouts with elastic load balancer in AWS
Vaibhav Puranik 2012-06-27, 22:03
Thanks Jun. How do I download 0.7.1?

I checked SVN tags but the last tag seems to be
kafka-0.7.1-incubating-candidate-3/<http://svn.apache.org/repos/asf/incubator/kafka/tags/kafka-0.7.1-incubating-candidate-3/>

Regards,
Vaibhav

On Wed, Jun 27, 2012 at 2:56 PM, Jun Rao <[EMAIL PROTECTED]> wrote:

> num.retries is added in 0.7.1, which is just out.
>
> Thanks,
>
> Jun
>
> On Wed, Jun 27, 2012 at 2:43 PM, Vaibhav Puranik <[EMAIL PROTECTED]>
> wrote:
>
> > Jun,
> >
> > I wrote a test producer to test if num.retries working or not. But I
> found
> > that it's not working. No matter how many retries I set,  whenever a
> > message send fails, it always never gets to the broker.
> > I am using Kafka 0.7.0
> >
> > Is this a known  problem? Do I need to file a JIRA issue?
> >
> > Because we are using Async producer we have no way to catch the exception
> > ourselves and act on it. Is that right? Any ideas how we can ensure that
> > every single message is sent with retries?
> >
> > Regards,
> > Vaibhav
> >
> > On Tue, Jun 26, 2012 at 4:52 PM, Jun Rao <[EMAIL PROTECTED]> wrote:
> >
> > > Set num.retries in producer config property file. It defaults to 0.
> > >
> > > Thanks,
> > >
> > > Jun
> > >
> > > On Tue, Jun 26, 2012 at 4:46 PM, Vaibhav Puranik <[EMAIL PROTECTED]>
> > > wrote:
> > >
> > > > I reduced the batch size and reduced the pooled connections. Number
> of
> > > > errors have gone down significantly. But they are not eliminated yet.
> > > >
> > > > We definitely don't want to loose any events.
> > > >
> > > > Jun, how do I configure the client resend you mentioned below? I
> > couldn't
> > > > find any configuration.
> > > >
> > > > Regards,
> > > > Vaibhav
> > > >
> > > > On Tue, Jun 26, 2012 at 9:27 AM, Vaibhav Puranik <[EMAIL PROTECTED]
> >
> > > > wrote:
> > > >
> > > > > These are great pointers.
> > > > > I found some more discussion here:
> > > > > https://forums.aws.amazon.com/thread.jspa?threadID=33427
> > > > >
> > > > > I can do the following to keep using the elastic load balancer:
> > > > >
> > > > > 1) Reduce the producer pool size to 1 or 2 because looks like
> > > connections
> > > > > are sitting idle. My volume does not desire that big pool.
> > > > > 2) Reduce the batch size so that the webapp frequently dumps the
> data
> > > to
> > > > > brokers. It's better for us anyways.
> > > > >
> > > > > I will try both of these options and report back.
> > > > >
> > > > > Thank you very much Jun and Niek.
> > > > >
> > > > > Regards,
> > > > > Vaibhav
> > > > >
> > > > >
> > > > > On Tue, Jun 26, 2012 at 8:52 AM, Niek Sanders <
> > [EMAIL PROTECTED]
> > > > >wrote:
> > > > >
> > > > >> ELBs will close connections that have no data going across them
> > over a
> > > > >> 60 sec period.  A reference to this behavior can be found at the
> > > > >> bottom of this page:
> > > > >>
> > > > >> http://aws.amazon.com/articles/1636185810492479
> > > > >>
> > > > >> There is currently no way for customers to increase this timeout.
> >  If
> > > > >> this timeout is in fact the problem, then the alternative is to
> use
> > HA
> > > > >> proxy for load balancing instead.
> > > > >>
> > > > >> - Niek
> > > > >>
> > > > >>
> > > > >>
> > > > >>
> > > > >> On Tue, Jun 26, 2012 at 7:55 AM, Jun Rao <[EMAIL PROTECTED]>
> wrote:
> > > > >> > Vaibhav,
> > > > >> >
> > > > >> > Does elastic load balancer have any timeouts or quotas that kill
> > > > >> existing
> > > > >> > socket connections? Does client resend succeed (you can
> configure
> > > > >> resend in
> > > > >> > DefaultEventHandler)?
> > > > >> >
> > > > >> > Thanks,
> > > > >> >
> > > > >> > Jun
> > > > >> >
> > > > >> > On Mon, Jun 25, 2012 at 6:01 PM, Vaibhav Puranik <
> > > [EMAIL PROTECTED]>
> > > > >> wrote:
> > > > >> >
> > > > >> >> Hi all,
> > > > >> >>
> > > > >> >> We are sending our ad impressions to Kafka 0.7.0. I am using
> > async
> > > > >> >> prouducers in our web app.
> > > > >> >> I am pooling kafak producers with commons pool. Pool size - 10.
> >