Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # user >> Purgatory


Copy link to this message
-
Re: Purgatory
Priya, if you want you can look at RequestPurgatory.scala for some more
details.

The config is the size of the atomic requestCounter.

Basically the purge in the purgatory is a way to check if the request has
been satisfied and delayed and can get removed.  It is a background scan
when the size reaches a certain point to get it back low again if it can
force it.  Kind of like a minor compaction how I see it so it does not grow
without bound.

/*******************************************
 Joe Stein
 Founder, Principal Consultant
 Big Data Open Source Security LLC
 http://www.stealth.ly
 Twitter: @allthingshadoop <http://www.twitter.com/allthingshadoop>
********************************************/
On Fri, Nov 1, 2013 at 2:45 AM, Priya Matpadi
<[EMAIL PROTECTED]>wrote:

> Guozhang,
> The documentation is not very clear.
> Marc's response for producer purgatory makes sense.
> I am not entirely clear on fetch purgatory.
> How does broker use purgatory? Is it a temporary holding area? What happens
> to the messages if purge interval is exceeded in case of either/both
> producer and consumer? Are messages dropped in this case?
> Thanks,
> Priya
>
>
> On Thu, Oct 31, 2013 at 2:47 PM, Guozhang Wang <[EMAIL PROTECTED]> wrote:
>
> > Hello Priya,
> >
> > You can find the definitions of these two configs here:
> >
> > http://kafka.apache.org/documentation.html#brokerconfigs
> >
> > Guozhang
> >
> >
> > On Thu, Oct 31, 2013 at 11:20 AM, Marc Labbe <[EMAIL PROTECTED]> wrote:
> >
> > > Hi Priya
> > >
> > > my understanding is producer requests will be delayed (and put in
> request
> > > purgatory) only if your producer uses ack=-1. It will be in the
> purgatory
> > > (delayed) until all brokers have acknowledged the messages to be
> > > replicated. The documentation suggests to monitor the
> > > ProducerRequestPurgatory size metrics , but it only applies if you're
> > using
> > > ack=-1, otherwise, this value will always be 0.
> > >
> > > For consumer requests, they'll be in purgatory (delayed) until the max
> > > allowed time to respond has been reached, unless it has enough messages
> > to
> > > fill the buffer before that. The request will not end up in the
> purgatory
> > > if you're making a blocking request (max wait <= 0).
> > >
> > > Not sure about the configuration interval though.
> > >
> > > marc
> > >
> > >
> > > On Thu, Oct 31, 2013 at 12:41 PM, Priya Matpadi <
> > > [EMAIL PROTECTED]
> > > > wrote:
> > >
> > > > Hello,
> > > > What is purgatory? I believe the following two properties relate to
> > > > consumer and producer respectively.
> > > > Could someone please explain the significance of these?
> > > > fetch.purgatory.purge.interval.requests=100
> > > > producer.purgatory.purge.interval.requests=100
> > > >
> > > > Thanks,
> > > > Priya
> > > >
> > >
> >
> >
> >
> > --
> > -- Guozhang
> >
>