Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # dev >> [jira] [Commented] (KAFKA-656) Add Quotas to Kafka


Copy link to this message
-
Re: [jira] [Commented] (KAFKA-656) Add Quotas to Kafka
This thread might be of some use, it discusses the memory footprint when
using metrics:
https://groups.google.com/forum/#!topic/metrics-user/T68H70ea1Og

On Sat, Jul 13, 2013 at 5:35 PM, Swapnil Ghike (JIRA) <[EMAIL PROTECTED]>wrote:

>
>     [
> https://issues.apache.org/jira/browse/KAFKA-656?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13707836#comment-13707836]
>
> Swapnil Ghike commented on KAFKA-656:
> -------------------------------------
>
> Hey Prashanth/Jonathan, have you had the time to run with this? If you're
> busy, I would be interested in taking this up.
>
> > Add Quotas to Kafka
> > -------------------
> >
> >                 Key: KAFKA-656
> >                 URL: https://issues.apache.org/jira/browse/KAFKA-656
> >             Project: Kafka
> >          Issue Type: New Feature
> >          Components: core
> >    Affects Versions: 0.8.1
> >            Reporter: Jay Kreps
> >              Labels: project
> >
> > It would be nice to implement a quota system in Kafka to improve our
> support for highly multi-tenant usage. The goal of this system would be to
> prevent one naughty user from accidently overloading the whole cluster.
> > There are several quantities we would want to track:
> > 1. Requests pers second
> > 2. Bytes written per second
> > 3. Bytes read per second
> > There are two reasonable groupings we would want to aggregate and
> enforce these thresholds at:
> > 1. Topic level
> > 2. Client level (e.g. by client id from the request)
> > When a request hits one of these limits we will simply reject it with a
> QUOTA_EXCEEDED exception.
> > To avoid suddenly breaking things without warning, we should ideally
> support two thresholds: a soft threshold at which we produce some kind of
> warning and a hard threshold at which we give the error. The soft threshold
> could just be defined as 80% (or whatever) of the hard threshold.
> > There are nuances to getting this right. If you measure second-by-second
> a single burst may exceed the threshold, so we need a sustained measurement
> over a period of time.
> > Likewise when do we stop giving this error? To make this work right we
> likely need to charge against the quota for request *attempts* not just
> successful requests. Otherwise a client that is overloading the server will
> just flap on and off--i.e. we would disable them for a period of time but
> when we re-enabled them they would likely still be abusing us.
> > It would be good to a wiki design on how this would all work as a
> starting point for discussion.
>
> --
> This message is automatically generated by JIRA.
> If you think it was sent incorrectly, please contact your JIRA
> administrators
> For more information on JIRA, see: http://www.atlassian.com/software/jira
>