Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # user >> Consumer throughput imbalance


Copy link to this message
-
Re: Consumer throughput imbalance
I'm still a little confused by your description of the problem. It might be
easier to understand if you listed out the exact things you have measured,
what you saw, and what you expected to see.

Since you mentioned the consumer I can give a little info on how that
works. The consumer consumes from all the partitions it owns
simultaneously. The behavior is that we interleve fetched data chunks of
messages from each partition the consumer is processing. The chunk size is
controlled by the fetch size set in the consumer. So the behavior you would
expect is that you would get a bunch of messages from one partition
followed by a bunch from another partition. The reason for doing this
instead of, say, interleving individual messages is that it is a big
performance boost--making every message an entry in a blocking queue gives
a 5x performance hit in high-throughput cases. Perhaps this interleaving is
the problem?

-Jay
On Sun, Aug 25, 2013 at 10:22 AM, Ian Friedman <[EMAIL PROTECTED]> wrote:

> Sorry I reread what I've written so far and found that it doesn't state
> the actual problem very well. Let me clarify once again:
>
> The problem we're trying to solve is that we can't let messages go for
> unbounded amounts of time without getting processed, and it seems that
> something about what we're doing (which I suspect is the fact that
> consumers own several partitions but only consume from one of them at a
> time until it's caught up) is causing a small number of them to sit around
> for hours and hours. This is despite some consumers idling due to being
> fully caught up on the partitions they own. We've found that requeueing the
> oldest messages (consumers ignore messages that have already been
> processed) is fairly effective in getting them to go away, but I'm looking
> for a more stable solution.
>
> --
> Ian Friedman
>
>
> On Sunday, August 25, 2013 at 1:15 PM, Ian Friedman wrote:
>
> > When I said "some messages take longer than others" that may have been
> misleading. What I meant there is that the performance of the entire
> application is inconsistent, mostly due to pressure from other applications
> (mapreduce) on our HBase and MySQL backends. On top of that, some messages
> just contain more data. Now I suppose what you're suggesting is that I
> segment my messages by the average or expected time it takes the payloads
> to process, but I suspect what will happen if I do that is I will have
> several consumers doing nothing most of the time, and the rest of them
> backlogged inconsistently the same way they are now. The problem isn't so
> much the size of the payloads but the fact that we're seeing some messages,
> which i suspect are in partitions with lots of longer running processing
> tasks, sit around for hours without getting consumed. That's what I'm
> trying to solve.
> >
> > Is there any way to "add more consumers" without actually adding more
> consumer JVM processes? We've hit something of a saturation point for our
> MySQL database. Is this maybe where having multiple consumer threads would
> help? If so, given that I have a singular shared processing queue in each
> consumer, how would I leverage that to solve this problem?
> >
> > --
> > Ian Friedman
> >
> >
> > On Sunday, August 25, 2013 at 12:13 PM, Mark wrote:
> >
> > > I don't think it would matter as long as you separate the types of
> message in different topics. Then just add more consumers to the ones that
> are slow. Am I missing something?
> > >
> > > On Aug 25, 2013, at 8:59 AM, Ian Friedman <[EMAIL PROTECTED] (mailto:
> [EMAIL PROTECTED])> wrote:
> > >
> > > > What if you don't know ahead of time how long a message will take to
> consume?
> > > >
> > > > --
> > > > Ian Friedman
> > > >
> > > >
> > > > On Sunday, August 25, 2013 at 10:45 AM, Neha Narkhede wrote:
> > > >
> > > > > Making producer side partitioning depend on consumer behavior
> might not be
> > > > > such a good idea. If consumption is a bottleneck, changing
> producer side
> > > > > partitioning may not help. To relieve consumption bottleneck, you

 
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB