Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # user >> Purgatory


Copy link to this message
-
Re: Purgatory
Priya, if you want you can look at RequestPurgatory.scala for some more
details.

The config is the size of the atomic requestCounter.

Basically the purge in the purgatory is a way to check if the request has
been satisfied and delayed and can get removed.  It is a background scan
when the size reaches a certain point to get it back low again if it can
force it.  Kind of like a minor compaction how I see it so it does not grow
without bound.

/*******************************************
 Joe Stein
 Founder, Principal Consultant
 Big Data Open Source Security LLC
 http://www.stealth.ly
 Twitter: @allthingshadoop <http://www.twitter.com/allthingshadoop>
********************************************/
On Fri, Nov 1, 2013 at 2:45 AM, Priya Matpadi
<[EMAIL PROTECTED]>wrote:

> Guozhang,
> The documentation is not very clear.
> Marc's response for producer purgatory makes sense.
> I am not entirely clear on fetch purgatory.
> How does broker use purgatory? Is it a temporary holding area? What happens
> to the messages if purge interval is exceeded in case of either/both
> producer and consumer? Are messages dropped in this case?
> Thanks,
> Priya
>
>
> On Thu, Oct 31, 2013 at 2:47 PM, Guozhang Wang <[EMAIL PROTECTED]> wrote:
>
> > Hello Priya,
> >
> > You can find the definitions of these two configs here:
> >
> > http://kafka.apache.org/documentation.html#brokerconfigs
> >
> > Guozhang
> >
> >
> > On Thu, Oct 31, 2013 at 11:20 AM, Marc Labbe <[EMAIL PROTECTED]> wrote:
> >
> > > Hi Priya
> > >
> > > my understanding is producer requests will be delayed (and put in
> request
> > > purgatory) only if your producer uses ack=-1. It will be in the
> purgatory
> > > (delayed) until all brokers have acknowledged the messages to be
> > > replicated. The documentation suggests to monitor the
> > > ProducerRequestPurgatory size metrics , but it only applies if you're
> > using
> > > ack=-1, otherwise, this value will always be 0.
> > >
> > > For consumer requests, they'll be in purgatory (delayed) until the max
> > > allowed time to respond has been reached, unless it has enough messages
> > to
> > > fill the buffer before that. The request will not end up in the
> purgatory
> > > if you're making a blocking request (max wait <= 0).
> > >
> > > Not sure about the configuration interval though.
> > >
> > > marc
> > >
> > >
> > > On Thu, Oct 31, 2013 at 12:41 PM, Priya Matpadi <
> > > [EMAIL PROTECTED]
> > > > wrote:
> > >
> > > > Hello,
> > > > What is purgatory? I believe the following two properties relate to
> > > > consumer and producer respectively.
> > > > Could someone please explain the significance of these?
> > > > fetch.purgatory.purge.interval.requests=100
> > > > producer.purgatory.purge.interval.requests=100
> > > >
> > > > Thanks,
> > > > Priya
> > > >
> > >
> >
> >
> >
> > --
> > -- Guozhang
> >
>

 
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB