Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Kafka, mail # user - Analysis of producer performance


+
Piotr Kozikowski 2013-04-08, 23:43
+
Jun Rao 2013-04-09, 04:49
+
Guy Doulberg 2013-04-09, 06:34
+
Piotr Kozikowski 2013-04-09, 17:23
+
Otis Gospodnetic 2013-04-10, 19:05
+
Piotr Kozikowski 2013-04-10, 20:11
Copy link to this message
-
Re: Analysis of producer performance
Yiu Wing TSANG 2013-04-11, 02:47
Piotr,

Thanks for your posts and after your comments about the need of "spooling",
I just find out this link:

http://grokbase.com/t/kafka/dev/133939nbvg/jira-commented-kafka-156-messages-should-not-be-dropped-when-brokers-are-unavailable

You are right that we need to do the spooling system by ourself until the
above issue is fixed.

I wonder if everybody is doing this spooling system by themselves?

As I just pushed kafka to our production system for "non-critical"
purposes, but eventually I want to make sure kafka will not lose message,
any idea to share to do this spooling system?

Wing
On Thu, Apr 11, 2013 at 4:10 AM, Piotr Kozikowski <[EMAIL PROTECTED]>wrote:

> Otis,
>
> That's actually a question we are trying to answer. In our current
> production system, Scribe does spooling to local disk, so each producer
> node becomes a local broker until the actual brokers are able to receive
> all messages again. It looks like unless a similar feature is added to
> Kafka we will have to come up with our own spooling system.
>
> -Piotr
>
> On Wed, Apr 10, 2013 at 12:04 PM, Otis Gospodnetic <
> [EMAIL PROTECTED]> wrote:
>
> > Hi,
> >
> > Is there anything one can do to "defend" from:
> >
> > "Trying to push more data than the brokers can handle for any sustained
> > period of time has catastrophic consequences, regardless of what timeout
> > settings are used. In our use case this means that we need to either
> ensure
> > we have spare capacity for spikes, or use something on top of Kafka to
> > absorb spikes."
> >
> > ?
> > Thanks,
> > Otis
> > ----
> > Performance Monitoring for Solr / ElasticSearch / HBase -
> > http://sematext.com/spm
> >
> >
> >
> >
> >
> > >________________________________
> > > From: Piotr Kozikowski <[EMAIL PROTECTED]>
> > >To: [EMAIL PROTECTED]
> > >Sent: Tuesday, April 9, 2013 1:23 PM
> > >Subject: Re: Analysis of producer performance
> > >
> > >Jun,
> > >
> > >Thank you for your comments. I'll reply point by point for clarity.
> > >
> > >1. We were aware of the migration tool but since we haven't used Kafka
> for
> > >production yet we just started using the 0.8 version directly.
> > >
> > >2. I hadn't seen those particular slides, very interesting. I'm not sure
> > >we're testing the same thing though. In our case we vary the number of
> > >physical machines, but each one has 10 threads accessing a pool of Kafka
> > >producer objects and in theory a single machine is enough to saturate
> the
> > >brokers (which our test mostly confirms). Also, assuming that the slides
> > >are based on the built-in producer performance tool, I know that we
> > started
> > >getting very different numbers once we switched to use "real" (actual
> > >production log) messages. Compression may also be a factor in case it
> > >wasn't configured the same way in those tests.
> > >
> > >3. In the latency section, there are two tests, one for average and
> > another
> > >for maximum latency. Each one has two graphs presenting the exact same
> > data
> > >but at different levels of zoom. The first one is to observe small
> > >variations of latency when target throughput <= actual throughput. The
> > >second is to observe the overall shape of the graph once latency starts
> > >growing when target throughput > actual throughput. I hope that makes
> > sense.
> > >
> > >4. That sounds great, looking forward to it.
> > >
> > >Piotr
> > >
> > >On Mon, Apr 8, 2013 at 9:48 PM, Jun Rao <[EMAIL PROTECTED]> wrote:
> > >
> > >> Piotr,
> > >>
> > >> Thanks for sharing this. Very interesting and useful study. A few
> > comments:
> > >>
> > >> 1. For existing 0.7 users, we have a migration tool that mirrors data
> > from
> > >> an 0.7 cluster to an 0.8 cluster. Applications can upgrade to 0.8 by
> > >> upgrading consumers first, followed by producers.
> > >>
> > >> 2. Have you looked at the Kafka ApacheCon slides (
> > >> http://www.slideshare.net/junrao/kafka-replication-apachecon2013)?
> > Towards
> > >> the end, there are some performance numbers too. The figure for

 
+
Jun Rao 2013-04-11, 05:18
+
Piotr Kozikowski 2013-04-12, 00:46
+
Jun Rao 2013-04-12, 14:54
+
Piotr Kozikowski 2013-04-12, 23:09
+
Jun Rao 2013-04-15, 01:06
+
Philip OToole 2013-04-12, 15:22
+
S Ahmed 2013-04-12, 15:28
+
Philip OToole 2013-04-12, 15:59
+
Philip OToole 2013-04-12, 17:04
+
Piotr Kozikowski 2013-04-15, 18:19
+
David Arthur 2013-04-23, 12:22