Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # user >> Getting timeouts with elastic load balancer in AWS


Copy link to this message
-
Re: Getting timeouts with elastic load balancer in AWS
Thanks Neha.

I will try num.retries again with this version and post my feedback here.

Regards,
Vaibhav

On Wed, Jun 27, 2012 at 3:13 PM, Neha Narkhede <[EMAIL PROTECTED]>wrote:

> You can download it from here -
>
> https://www.apache.org/dyn/closer.cgi/incubator/kafka/kafka-0.7.1-incubating/
>
> Thanks,
> Neha
>
> On Wed, Jun 27, 2012 at 3:03 PM, Vaibhav Puranik <[EMAIL PROTECTED]>
> wrote:
> > Thanks Jun. How do I download 0.7.1?
> >
> > I checked SVN tags but the last tag seems to be
> > kafka-0.7.1-incubating-candidate-3/<
> http://svn.apache.org/repos/asf/incubator/kafka/tags/kafka-0.7.1-incubating-candidate-3/
> >
> >
> > Regards,
> > Vaibhav
> >
> > On Wed, Jun 27, 2012 at 2:56 PM, Jun Rao <[EMAIL PROTECTED]> wrote:
> >
> >> num.retries is added in 0.7.1, which is just out.
> >>
> >> Thanks,
> >>
> >> Jun
> >>
> >> On Wed, Jun 27, 2012 at 2:43 PM, Vaibhav Puranik <[EMAIL PROTECTED]>
> >> wrote:
> >>
> >> > Jun,
> >> >
> >> > I wrote a test producer to test if num.retries working or not. But I
> >> found
> >> > that it's not working. No matter how many retries I set,  whenever a
> >> > message send fails, it always never gets to the broker.
> >> > I am using Kafka 0.7.0
> >> >
> >> > Is this a known  problem? Do I need to file a JIRA issue?
> >> >
> >> > Because we are using Async producer we have no way to catch the
> exception
> >> > ourselves and act on it. Is that right? Any ideas how we can ensure
> that
> >> > every single message is sent with retries?
> >> >
> >> > Regards,
> >> > Vaibhav
> >> >
> >> > On Tue, Jun 26, 2012 at 4:52 PM, Jun Rao <[EMAIL PROTECTED]> wrote:
> >> >
> >> > > Set num.retries in producer config property file. It defaults to 0.
> >> > >
> >> > > Thanks,
> >> > >
> >> > > Jun
> >> > >
> >> > > On Tue, Jun 26, 2012 at 4:46 PM, Vaibhav Puranik <
> [EMAIL PROTECTED]>
> >> > > wrote:
> >> > >
> >> > > > I reduced the batch size and reduced the pooled connections.
> Number
> >> of
> >> > > > errors have gone down significantly. But they are not eliminated
> yet.
> >> > > >
> >> > > > We definitely don't want to loose any events.
> >> > > >
> >> > > > Jun, how do I configure the client resend you mentioned below? I
> >> > couldn't
> >> > > > find any configuration.
> >> > > >
> >> > > > Regards,
> >> > > > Vaibhav
> >> > > >
> >> > > > On Tue, Jun 26, 2012 at 9:27 AM, Vaibhav Puranik <
> [EMAIL PROTECTED]
> >> >
> >> > > > wrote:
> >> > > >
> >> > > > > These are great pointers.
> >> > > > > I found some more discussion here:
> >> > > > > https://forums.aws.amazon.com/thread.jspa?threadID=33427
> >> > > > >
> >> > > > > I can do the following to keep using the elastic load balancer:
> >> > > > >
> >> > > > > 1) Reduce the producer pool size to 1 or 2 because looks like
> >> > > connections
> >> > > > > are sitting idle. My volume does not desire that big pool.
> >> > > > > 2) Reduce the batch size so that the webapp frequently dumps the
> >> data
> >> > > to
> >> > > > > brokers. It's better for us anyways.
> >> > > > >
> >> > > > > I will try both of these options and report back.
> >> > > > >
> >> > > > > Thank you very much Jun and Niek.
> >> > > > >
> >> > > > > Regards,
> >> > > > > Vaibhav
> >> > > > >
> >> > > > >
> >> > > > > On Tue, Jun 26, 2012 at 8:52 AM, Niek Sanders <
> >> > [EMAIL PROTECTED]
> >> > > > >wrote:
> >> > > > >
> >> > > > >> ELBs will close connections that have no data going across them
> >> > over a
> >> > > > >> 60 sec period.  A reference to this behavior can be found at
> the
> >> > > > >> bottom of this page:
> >> > > > >>
> >> > > > >> http://aws.amazon.com/articles/1636185810492479
> >> > > > >>
> >> > > > >> There is currently no way for customers to increase this
> timeout.
> >> >  If
> >> > > > >> this timeout is in fact the problem, then the alternative is to
> >> use
> >> > HA
> >> > > > >> proxy for load balancing instead.
> >> > > > >>
> >> > > > >> - Niek
> >> > > > >>
> >> > > > >>
> >> > > > >>
> >> > > > >>
> >> > > > >> On Tue, Jun 26, 2012 at 7:55 AM, Jun Rao <[EMAIL PROTECTED]>
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB