Swapnil Ghike 2013-07-13, 21:35
-Re: [jira] [Commented] (KAFKA-656) Add Quotas to Kafka
S Ahmed 2013-07-14, 19:09
This thread might be of some use, it discusses the memory footprint when
On Sat, Jul 13, 2013 at 5:35 PM, Swapnil Ghike (JIRA) <[EMAIL PROTECTED]>wrote:
> Swapnil Ghike commented on KAFKA-656:
> Hey Prashanth/Jonathan, have you had the time to run with this? If you're
> busy, I would be interested in taking this up.
> > Add Quotas to Kafka
> > -------------------
> > Key: KAFKA-656
> > URL: https://issues.apache.org/jira/browse/KAFKA-656
> > Project: Kafka
> > Issue Type: New Feature
> > Components: core
> > Affects Versions: 0.8.1
> > Reporter: Jay Kreps
> > Labels: project
> > It would be nice to implement a quota system in Kafka to improve our
> support for highly multi-tenant usage. The goal of this system would be to
> prevent one naughty user from accidently overloading the whole cluster.
> > There are several quantities we would want to track:
> > 1. Requests pers second
> > 2. Bytes written per second
> > 3. Bytes read per second
> > There are two reasonable groupings we would want to aggregate and
> enforce these thresholds at:
> > 1. Topic level
> > 2. Client level (e.g. by client id from the request)
> > When a request hits one of these limits we will simply reject it with a
> QUOTA_EXCEEDED exception.
> > To avoid suddenly breaking things without warning, we should ideally
> support two thresholds: a soft threshold at which we produce some kind of
> warning and a hard threshold at which we give the error. The soft threshold
> could just be defined as 80% (or whatever) of the hard threshold.
> > There are nuances to getting this right. If you measure second-by-second
> a single burst may exceed the threshold, so we need a sustained measurement
> over a period of time.
> > Likewise when do we stop giving this error? To make this work right we
> likely need to charge against the quota for request *attempts* not just
> successful requests. Otherwise a client that is overloading the server will
> just flap on and off--i.e. we would disable them for a period of time but
> when we re-enabled them they would likely still be abusing us.
> > It would be good to a wiki design on how this would all work as a
> starting point for discussion.
> This message is automatically generated by JIRA.
> If you think it was sent incorrectly, please contact your JIRA
> For more information on JIRA, see: http://www.atlassian.com/software/jira