Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # user >> seeing poor consumer performance in 0.7.2


Copy link to this message
-
Re: seeing poor consumer performance in 0.7.2
Is your network shared? Is so, another possibility is that some other apps
are consuming the bandwidth.

Thanks,

Jun
On Sun, Apr 21, 2013 at 12:23 PM, Andrew Neilson <[EMAIL PROTECTED]>wrote:

> Thanks very much for the reply Neha! So I swapped out the consumer that
> processes the messages with one that just prints them. It does indeed
> achieve a much better rate at peaks but can still nearly zero out (if not
> completely zero out). I plotted the messages printed in graphite to show
> the behaviour I'm seeing (this is messages printed per second):
>
>
> https://www.dropbox.com/s/7u7uyrefw6inetu/Screen%20Shot%202013-04-21%20at%2011.44.38%20AM.png
>
> The peaks are over ten thousand per second and the troughs can go below 10
> per second just prior to another peak. I know that there are plenty of
> messages available because the ones currently being processed are still
> from Friday afternoon, so this may or may not have something to do with
> this pattern.
>
> Is there anything I can do to avoid the periods of lower performance?
> Ideally I would be processing messages as soon as they are written.
>
>
> On Sun, Apr 21, 2013 at 8:49 AM, Neha Narkhede <[EMAIL PROTECTED]
> >wrote:
>
> > Some of the reasons a consumer is slow are -
> > 1. Small fetch size
> > 2. Expensive message processing
> >
> > Are you processing the received messages in the consumer ? Have you
> > tried running console consumer for this topic and see how it performs
> > ?
> >
> > Thanks,
> > Neha
> >
> > On Sun, Apr 21, 2013 at 1:59 AM, Andrew Neilson <[EMAIL PROTECTED]>
> > wrote:
> > > I am currently running a deployment with 3 brokers, 3 ZK, 3 producers,
> 2
> > > consumers, and 15 topics. I should first point out that this is my
> first
> > > project using Kafka ;). The issue I'm seeing is that the consumers are
> > only
> > > processing about 15 messages per second from what should be the largest
> > > topic it is consuming (we're sending 200-400 ~300 byte messages per
> > second
> > > to this topic). I should note that I'm using a high level ZK consumer
> and
> > > ZK 3.4.3.
> > >
> > > I have a strong feeling I have not configured things properly so I
> could
> > > definitely use some guidance. Here is my broker configuration:
> > >
> > > brokerid=1
> > > port=9092
> > > socket.send.buffer=1048576
> > > socket.receive.buffer=1048576
> > > max.socket.request.bytes=104857600
> > > log.dir=/home/kafka/data
> > > num.partitions=1
> > > log.flush.interval=10000
> > > log.default.flush.interval.ms=1000
> > > log.default.flush.scheduler.interval.ms=1000
> > > log.retention.hours=168
> > > log.file.size=536870912
> > > enable.zookeeper=true
> > > zk.connect=XXX
> > > zk.connectiontimeout.ms=1000000
> > >
> > > Here is my producer config:
> > >
> > > zk.connect=XXX
> > > producer.type=async
> > > compression.codec=0
> > >
> > > Here is my consumer config:
> > >
> > > zk.connect=XXX
> > > zk.connectiontimeout.ms=100000
> > > groupid=XXX
> > > autooffset.reset=smallest
> > > socket.buffersize=1048576
> > > fetch.size=10485760
> > > queuedchunks.max=10000
> > >
> > > Thanks for any assistance you can provide,
> > >
> > > Andrew
> >
>