Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
HBase >> mail # dev >> high-percentile latency metrics, aka HBASE-6261


Copy link to this message
-
Re: high-percentile latency metrics, aka HBASE-6261
I put this on the jira too, but the algo I found whittled down a stream of
10 million items down to ~19.5k samples. With each sample at ~36B, that's
~685KiB. There's a bit more from using a LinkedList and general bookkeeping.

Since the estimator is reset every O(minutes) window, and I doubt very many
metrics see more than 10 million items in O(minutes), it seems lightweight
enough to keep going.

I'm planning on doing this in hadoop-common's metrics2 since HDFS is also
interested, backporting to 1.x and 2.x. This would thus depend on the
metrics2 conversion (HBASE-4050) going through too.

Thanks,
Andrew

On Thu, Jun 28, 2012 at 3:31 PM, Stack <[EMAIL PROTECTED]> wrote:

> On Tue, Jun 26, 2012 at 6:35 PM, Andrew Wang <[EMAIL PROTECTED]>
> wrote:
> > I wanted to ask off JIRA though about what would be useful in practice. I
> > think it'd be nice to see, for example, accurate 90th and 99th percentile
> > latency over recent 10s, 1m, 5m, and 15m time windows. I found some nice
> > algos to do this, I think at the cost of MBs of memory.
> >
>
> Agree.
>
> How many MBs?
>
>
> > So, is the "full" solution compelling enough to proceed? Anything
> > missing/extraneous?
> >
>
> Whats going on is a critical focus going forward so I'd say 'full'
> unless the cost obscene.
>
> St.Ack
>
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB