Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # user >> Purgatory


Copy link to this message
-
Re: Purgatory
To edit the Wiki you need to send an ICLA
http://www.apache.org/licenses/#clas to Apache and then once that is done
an email to [EMAIL PROTECTED] (or to me and I will copy private)
with your Wiki username and that you sent the ICLA to Apache.

Then, I can add you to edit the Wiki.

/*******************************************
 Joe Stein
 Founder, Principal Consultant
 Big Data Open Source Security LLC
 http://www.stealth.ly
 Twitter: @allthingshadoop <http://www.twitter.com/allthingshadoop>
********************************************/
On Fri, Nov 1, 2013 at 9:08 PM, Marc Labbe <[EMAIL PROTECTED]> wrote:

> Hi Joel,
>
> I used to have edit to the wiki, I made a few additions to it a while ago
> but it's seem I don't have it anymore. It might have been lost in the
> confluence update. I would be glad to add what I have written if I get it
> back. Otherwise, feel free to paste my words in one of the pages, I don't
> intend on asking for copyrights for this :).
>
> marc
>
>
> On Fri, Nov 1, 2013 at 4:32 PM, Joel Koshy <[EMAIL PROTECTED]> wrote:
>
> > Marc, thanks for writing that up. I think it is worth adding some
> > details on the request-purgatory on a wiki (Jay had started a wiki
> > page for kafka internals [1] a while ago, but we have not had time to
> > add much to it since.) Your write-up could be reviewed and added
> > there. Do you have edit permissions on the wiki?
> >
> > As for the purge interval config - yes the documentation can be
> > improved a bit. It's one of those "internal" configs that generally
> > don't need to be modified by users. The reason we added that was as
> > follows:
> > - We found that for low-volume topics, replica fetch requests were
> > getting expired but sitting around in purgatory
> > - This was because we were expiring them from the delay queue (used to
> > track when requests should expire), but they were still sitting in the
> > watcherFor map - i.e., they would get purged when the next producer
> > request to that topic/partition arrived, but for low volume topics
> > this could be a long time (or never in the worst case) and we would
> > eventually run into an OOME.
> > - So we needed to periodically go through the entire watcherFor map
> > and explicitly remove those requests that had expired.
> > - More details on this are in KAFKA-664.
> >
> > Thanks,
> >
> > Joel
> >
> > [1] https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Internals
> >
> > On Fri, Nov 1, 2013 at 12:33 PM, Marc Labbe <[EMAIL PROTECTED]> wrote:
> > > Guozhang,
> > >
> > > I have to agree with Priya the doc isn't very clear. Although the
> > > configuration is documented, it is simply rewording the name of the
> > config,
> > > which isn't particularly useful if you want more information about what
> > the
> > > purgatory is. I searched the whole wiki and doc and could not find
> > anything
> > > very useful as opposed looking a the code. In this case,
> > > kafka.server.KafkaApis and kafka.server.RequestPurgatory will be your
> > > friends.
> > >
> > > I'll try to add to Joe's answer here, mostly just reporting what's
> > > available in the Scala doc from the project. I am doing this to
> > understand
> > > the mechanics myself btw.
> > >
> > > As Joe said, messages are not dropped by the purgatory but simply
> removed
> > > from the purgatory when they are satisfied. Satisfaction conditions are
> > > different for both fetch and produce requests and this is implemented
> in
> > > their respective DelayedRequest implementation (DelayedFetch and
> > > DelayedProduce).
> > >
> > > Requests purgatories are defined as follow in the code:
> > >  - ProducerRequestPurgatory: A holding pen for produce requests waiting
> > to
> > > be satisfied.
> > >  - FetchRequestPurgatory: A holding pen for fetch requests waiting to
> be
> > > satisfied
> > >
> > > Each request purgatory runs a thread (ExpiredRequestReaper). This
> thread
> > > will first try to find an expired delayed request. When one if found,
> it

 
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB