I think what you are asking for is backpressure from the broker to the
producer. I.e. when the broker got close to full it would start to slow
down the producer to let the consumer catch up. This is a fairly typical
thing for a message broker to do.
Our approach is different though. We have found that most use cases cannot
tolerate backpressure because the production of messages comes from a live
service that cannot stop. So we have instead focused on scaling the data
that is retained and the consumption. In Kafka it is very reasonable to
retain 5TB per server and you can have lots of servers to provide
sufficient buffer. If you feel like there are use cases where backpressure
is preferrable it would probably be good for us to hear about them because
right now we haven't seen a need for it.
On Wed, Dec 11, 2013 at 11:42 PM, xingcan <[EMAIL PROTECTED]> wrote:
> Thanks for your prompt replay. I got two 300GB SAS disks for each borker.
> At peak time, the produce speed for each broker is about 70MB/s.
> this speed is already restricted by network. While, the consume speed is
> for some topics are consumed by more than one group. Under this
> if the peak time lasts for hours, my disks will be fully used.
> On Thu, Dec 12, 2013 at 2:00 PM, Guozhang Wang <[EMAIL PROTECTED]> wrote:
> > One possible approach is to change the retention policy on broker.
> > How large your messages can accumulate on brokers at peak time?
> > Guozhang
> > On Wed, Dec 11, 2013 at 9:09 PM, xingcan <[EMAIL PROTECTED]> wrote:
> > > Hi,
> > >
> > > In my application, the produce speed could be very high at some
> > > time in a day while return
> > > to a low speed at the rest of time. Frequently, my data logs are
> > > away before they are being
> > > consumed by clients due to lacking disk space during the busy times.
> > >
> > > Increasing consume speed seems to be difficult and adding space for log
> > > files are also not a ultimate solution. Any suggestions for this
> > > Are there any speed control mechanism in Kafka?
> > >
> > > Thanks.
> > >
> > > --
> > > *Xingcan*
> > >
> > --
> > -- Guozhang