Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # user >> Purgatory


Marc, thanks much for documenting the guts!
There is one correction for Fetch Request handling:

When is it satisfied?

The fetch size requested is reached - ie. the amount of data the consumer
wishes to receive in one response

Consumer configuration: *fetch.message.max.bytes*
As per the code:

  /**
   * A holding pen for fetch requests waiting to be satisfied
   */
  class FetchRequestPurgatory(requestChannel: RequestChannel,
purgeInterval: Int)
          extends RequestPurgatory[DelayedFetch, Int](brokerId,
purgeInterval) {
    this.logIdent = "[FetchRequestPurgatory-%d] ".format(brokerId)

    /**
     * A fetch request is satisfied when it has accumulated enough data to
meet the min_bytes field
     */
    def checkSatisfied(messageSizeInBytes: Int, delayedFetch:
DelayedFetch): Boolean = {
      val accumulatedSize =
delayedFetch.bytesAccumulated.addAndGet(messageSizeInBytes)
      accumulatedSize >= delayedFetch.fetch.minBytes
    }

On Fri, Nov 8, 2013 at 1:01 PM, Joel Koshy <[EMAIL PROTECTED]> wrote:

> Marc - thanks again for doing this.  Couple of suggestions:
>
> - I would suggest removing the disclaimer and email quotes since this
>   can become a stand-alone clean document on what the purgatory is and
>   how it works.
> - A diagram would be helpful - it could say, show the watcher map and
>   the expiration queue, and it will be especially useful if it can
>   show the flow of producer/fetch requests through the purgatory. That
>   would also help cut down a lot of the text in the doc.
> - I think it would be preferrable to have just high-level details in
>   this document.  Internal details (such as the purge interval
>   settings) can either be removed or moved (to say, a short faq or
>   config section at the end).
> - In the overview may want to comment on why we added it: i.e., it is
>   the primary data structure we use for supporting long poll of
>   producer/fetch requests. E.g., if we don't do this consumers would
>   have to keep issuing fetch requests if there's no data yet - as
>   opposed to just saying "respond when 'n' bytes of data are available
>   or when 't' millisecs have elapsed, whichever is earlier."
> - WRT your question on PurgatorySize - we added that just to keep a
>   tab on how many requests are sitting in purgatory (including both
>   watchers map and expiration queue) as a rough gauge of memory usage.
>   Also the fetch/producer request gauges should not collide - the
>   KafkaMetricsGroup class takes care of this. The CSV reporter might
>   run into issues though - I thought we had fixed that but could be
>   wrong.
>
> Joel
>
> On Thu, Nov 07, 2013 at 11:01:06PM -0800, Joel Koshy wrote:
> > Excellent - thanks for putting that together! Will review it more
> > carefully tomorrow and suggest some minor edits if required.
> >
> > On Thu, Nov 07, 2013 at 10:45:40PM -0500, Marc Labbe wrote:
> > > I've just added a page for purgatory, feel free to comment/modify at
> will.
> > > I hope I didn't misinterpret too much of the code.
> > >
> > >
> https://cwiki.apache.org/confluence/display/KAFKA/Request+Purgatory+(0.8)
> > >
> > > I added a few questions of my own.
> > >
> > >
> > > On Fri, Nov 1, 2013 at 9:43 PM, Joe Stein <[EMAIL PROTECTED]>
> wrote:
> > >
> > > > To edit the Wiki you need to send an ICLA
> > > > http://www.apache.org/licenses/#clas to Apache and then once that
> is done
> > > > an email to [EMAIL PROTECTED] (or to me and I will copy
> private)
> > > > with your Wiki username and that you sent the ICLA to Apache.
> > > >
> > > > Then, I can add you to edit the Wiki.
> > > >
> > > > /*******************************************
> > > >  Joe Stein
> > > >  Founder, Principal Consultant
> > > >  Big Data Open Source Security LLC
> > > >  http://www.stealth.ly
> > > >  Twitter: @allthingshadoop <http://www.twitter.com/allthingshadoop>
> > > > ********************************************/
> > > >
> > > >
> > > > On Fri, Nov 1, 2013 at 9:08 PM, Marc Labbe <[EMAIL PROTECTED]>