Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # user >> Purgatory


Copy link to this message
-
RE: Purgatory
Thx for the feedback. It is true I never mention anything about impact on
users or the fact this is mostly internal business in Kafka. I will try to
rephrase some of this.

Marc
On Nov 8, 2013 10:10 AM, "Yu, Libo" <[EMAIL PROTECTED]> wrote:

> I read it and tried to understand it. It would be great to add a summary
> at the beginning about what it is and how it may impact a user.
>
> Regards,
>
> Libo
>
>
> -----Original Message-----
> From: Joel Koshy [mailto:[EMAIL PROTECTED]]
> Sent: Friday, November 08, 2013 2:01 AM
> To: [EMAIL PROTECTED]
> Subject: Re: Purgatory
>
> Excellent - thanks for putting that together! Will review it more
> carefully tomorrow and suggest some minor edits if required.
>
> On Thu, Nov 07, 2013 at 10:45:40PM -0500, Marc Labbe wrote:
> > I've just added a page for purgatory, feel free to comment/modify at
> will.
> > I hope I didn't misinterpret too much of the code.
> >
> > https://cwiki.apache.org/confluence/display/KAFKA/Request+Purgatory+(0
> > .8)
> >
> > I added a few questions of my own.
> >
> >
> > On Fri, Nov 1, 2013 at 9:43 PM, Joe Stein <[EMAIL PROTECTED]> wrote:
> >
> > > To edit the Wiki you need to send an ICLA
> > > http://www.apache.org/licenses/#clas to Apache and then once that is
> > > done an email to [EMAIL PROTECTED] (or to me and I will copy
> > > private) with your Wiki username and that you sent the ICLA to Apache.
> > >
> > > Then, I can add you to edit the Wiki.
> > >
> > > /*******************************************
> > >  Joe Stein
> > >  Founder, Principal Consultant
> > >  Big Data Open Source Security LLC
> > >  http://www.stealth.ly
> > >  Twitter: @allthingshadoop <http://www.twitter.com/allthingshadoop>
> > > ********************************************/
> > >
> > >
> > > On Fri, Nov 1, 2013 at 9:08 PM, Marc Labbe <[EMAIL PROTECTED]> wrote:
> > >
> > > > Hi Joel,
> > > >
> > > > I used to have edit to the wiki, I made a few additions to it a
> > > > while ago but it's seem I don't have it anymore. It might have
> > > > been lost in the confluence update. I would be glad to add what I
> > > > have written if I get it back. Otherwise, feel free to paste my
> > > > words in one of the pages, I don't intend on asking for copyrights
> for this :).
> > > >
> > > > marc
> > > >
> > > >
> > > > On Fri, Nov 1, 2013 at 4:32 PM, Joel Koshy <[EMAIL PROTECTED]>
> wrote:
> > > >
> > > > > Marc, thanks for writing that up. I think it is worth adding
> > > > > some details on the request-purgatory on a wiki (Jay had started
> > > > > a wiki page for kafka internals [1] a while ago, but we have not
> > > > > had time to add much to it since.) Your write-up could be
> > > > > reviewed and added there. Do you have edit permissions on the wiki?
> > > > >
> > > > > As for the purge interval config - yes the documentation can be
> > > > > improved a bit. It's one of those "internal" configs that
> > > > > generally don't need to be modified by users. The reason we
> > > > > added that was as
> > > > > follows:
> > > > > - We found that for low-volume topics, replica fetch requests
> > > > > were getting expired but sitting around in purgatory
> > > > > - This was because we were expiring them from the delay queue
> > > > > (used to track when requests should expire), but they were still
> > > > > sitting in the watcherFor map - i.e., they would get purged when
> > > > > the next producer request to that topic/partition arrived, but
> > > > > for low volume topics this could be a long time (or never in the
> > > > > worst case) and we would eventually run into an OOME.
> > > > > - So we needed to periodically go through the entire watcherFor
> > > > > map and explicitly remove those requests that had expired.
> > > > > - More details on this are in KAFKA-664.
> > > > >
> > > > > Thanks,
> > > > >
> > > > > Joel
> > > > >
> > > > > [1]
> > > > > https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Internal
> > > > > s
> > > > >
> > > > > On Fri, Nov 1, 2013 at 12:33 PM, Marc Labbe <[EMAIL PROTECTED]>

 
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB