Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # user >> Purgatory


Guozhang,
The documentation is not very clear.
Marc's response for producer purgatory makes sense.
I am not entirely clear on fetch purgatory.
How does broker use purgatory? Is it a temporary holding area? What happens
to the messages if purge interval is exceeded in case of either/both
producer and consumer? Are messages dropped in this case?
Thanks,
Priya
On Thu, Oct 31, 2013 at 2:47 PM, Guozhang Wang <[EMAIL PROTECTED]> wrote:

> Hello Priya,
>
> You can find the definitions of these two configs here:
>
> http://kafka.apache.org/documentation.html#brokerconfigs
>
> Guozhang
>
>
> On Thu, Oct 31, 2013 at 11:20 AM, Marc Labbe <[EMAIL PROTECTED]> wrote:
>
> > Hi Priya
> >
> > my understanding is producer requests will be delayed (and put in request
> > purgatory) only if your producer uses ack=-1. It will be in the purgatory
> > (delayed) until all brokers have acknowledged the messages to be
> > replicated. The documentation suggests to monitor the
> > ProducerRequestPurgatory size metrics , but it only applies if you're
> using
> > ack=-1, otherwise, this value will always be 0.
> >
> > For consumer requests, they'll be in purgatory (delayed) until the max
> > allowed time to respond has been reached, unless it has enough messages
> to
> > fill the buffer before that. The request will not end up in the purgatory
> > if you're making a blocking request (max wait <= 0).
> >
> > Not sure about the configuration interval though.
> >
> > marc
> >
> >
> > On Thu, Oct 31, 2013 at 12:41 PM, Priya Matpadi <
> > [EMAIL PROTECTED]
> > > wrote:
> >
> > > Hello,
> > > What is purgatory? I believe the following two properties relate to
> > > consumer and producer respectively.
> > > Could someone please explain the significance of these?
> > > fetch.purgatory.purge.interval.requests=100
> > > producer.purgatory.purge.interval.requests=100
> > >
> > > Thanks,
> > > Priya
> > >
> >
>
>
>
> --
> -- Guozhang
>

 
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB