Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # user >> Purgatory


Copy link to this message
-
Re: Purgatory
To edit the Wiki you need to send an ICLA
http://www.apache.org/licenses/#clas to Apache and then once that is done
an email to [EMAIL PROTECTED] (or to me and I will copy private)
with your Wiki username and that you sent the ICLA to Apache.

Then, I can add you to edit the Wiki.

/*******************************************
 Joe Stein
 Founder, Principal Consultant
 Big Data Open Source Security LLC
 http://www.stealth.ly
 Twitter: @allthingshadoop <http://www.twitter.com/allthingshadoop>
********************************************/
On Fri, Nov 1, 2013 at 9:08 PM, Marc Labbe <[EMAIL PROTECTED]> wrote:

> Hi Joel,
>
> I used to have edit to the wiki, I made a few additions to it a while ago
> but it's seem I don't have it anymore. It might have been lost in the
> confluence update. I would be glad to add what I have written if I get it
> back. Otherwise, feel free to paste my words in one of the pages, I don't
> intend on asking for copyrights for this :).
>
> marc
>
>
> On Fri, Nov 1, 2013 at 4:32 PM, Joel Koshy <[EMAIL PROTECTED]> wrote:
>
> > Marc, thanks for writing that up. I think it is worth adding some
> > details on the request-purgatory on a wiki (Jay had started a wiki
> > page for kafka internals [1] a while ago, but we have not had time to
> > add much to it since.) Your write-up could be reviewed and added
> > there. Do you have edit permissions on the wiki?
> >
> > As for the purge interval config - yes the documentation can be
> > improved a bit. It's one of those "internal" configs that generally
> > don't need to be modified by users. The reason we added that was as
> > follows:
> > - We found that for low-volume topics, replica fetch requests were
> > getting expired but sitting around in purgatory
> > - This was because we were expiring them from the delay queue (used to
> > track when requests should expire), but they were still sitting in the
> > watcherFor map - i.e., they would get purged when the next producer
> > request to that topic/partition arrived, but for low volume topics
> > this could be a long time (or never in the worst case) and we would
> > eventually run into an OOME.
> > - So we needed to periodically go through the entire watcherFor map
> > and explicitly remove those requests that had expired.
> > - More details on this are in KAFKA-664.
> >
> > Thanks,
> >
> > Joel
> >
> > [1] https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Internals
> >
> > On Fri, Nov 1, 2013 at 12:33 PM, Marc Labbe <[EMAIL PROTECTED]> wrote:
> > > Guozhang,
> > >
> > > I have to agree with Priya the doc isn't very clear. Although the
> > > configuration is documented, it is simply rewording the name of the
> > config,
> > > which isn't particularly useful if you want more information about what
> > the
> > > purgatory is. I searched the whole wiki and doc and could not find
> > anything
> > > very useful as opposed looking a the code. In this case,
> > > kafka.server.KafkaApis and kafka.server.RequestPurgatory will be your
> > > friends.
> > >
> > > I'll try to add to Joe's answer here, mostly just reporting what's
> > > available in the Scala doc from the project. I am doing this to
> > understand
> > > the mechanics myself btw.
> > >
> > > As Joe said, messages are not dropped by the purgatory but simply
> removed
> > > from the purgatory when they are satisfied. Satisfaction conditions are
> > > different for both fetch and produce requests and this is implemented
> in
> > > their respective DelayedRequest implementation (DelayedFetch and
> > > DelayedProduce).
> > >
> > > Requests purgatories are defined as follow in the code:
> > >  - ProducerRequestPurgatory: A holding pen for produce requests waiting
> > to
> > > be satisfied.
> > >  - FetchRequestPurgatory: A holding pen for fetch requests waiting to
> be
> > > satisfied
> > >
> > > Each request purgatory runs a thread (ExpiredRequestReaper). This
> thread
> > > will first try to find an expired delayed request. When one if found,
> it