Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # user >> About speed contorl in Kafka


Copy link to this message
-
Re: About speed contorl in Kafka
Sounds to me your consumer performance is the problem, not the producer.
So either make your consumers faster or make them consume but drop messages
to keep up with the producer speed.
This also gives you some means of keeping track of how many messages are
lost, why, and when.

And just for reference; the Kafka broker->consumer throughput is far higher
than the producer->broker throughput.
2013/12/12 xingcan <[EMAIL PROTECTED]>

> Guozhang,
>
> Thanks for your prompt replay. I got two 300GB SAS disks for each borker.
> At peak time, the produce speed for each broker is about 70MB/s.
> Apparently,
> this speed is already restricted by network. While, the consume speed is
> lower
> for some topics are consumed by more than one group. Under this
> circumstance,
> if the peak time lasts for hours, my disks will be fully used.
>
>
> On Thu, Dec 12, 2013 at 2:00 PM, Guozhang Wang <[EMAIL PROTECTED]> wrote:
>
> > One possible approach is to change the retention policy on broker.
> >
> > How large your messages can accumulate on brokers at peak time?
> >
> > Guozhang
> >
> >
> > On Wed, Dec 11, 2013 at 9:09 PM, xingcan <[EMAIL PROTECTED]> wrote:
> >
> > > Hi,
> > >
> > > In my application, the produce speed could be very high at some
> specific
> > > time in a day while return
> > > to a low speed at the rest of time. Frequently, my data logs are
> flushed
> > > away before they are being
> > > consumed by clients due to lacking disk space during the busy times.
> > >
> > > Increasing consume speed seems to be difficult and adding space for log
> > > files are also not a ultimate solution. Any suggestions for this
> problem?
> > > Are there any speed control mechanism in Kafka?
> > >
> > > Thanks.
> > >
> > > --
> > > *Xingcan*
> > >
> >
> >
> >
> > --
> > -- Guozhang
> >
>
>
>
> --
> *Xingcan*
>

 
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB