Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # user >> Monitoring with Riemann


Copy link to this message
-
Re: Monitoring with Riemann
Paul,

Thanks for sharing.

Jun
On Thu, Jul 18, 2013 at 2:00 PM, Paul Ingles <[EMAIL PROTECTED]> wrote:

> Hi,
>
> Earlier today I finished work on a library that adds an additional metrics
> reporter (https://github.com/pingles/kafka-riemann-reporter). I did it
> because although we were writing the CSV metrics we had nothing to display
> the data.
>
> It pushes metrics to Riemann (http://riemann.io) a very cool tool for
> aggregating and processing event streams. We've experimented with it a
> little in the past but this is the first production integration we've had.
> Streams are processed by Riemann and then plotted by Librato (
> http://librato.com).
>
> Installing requires building the project and copying the JAR to Kafka's
> ./libs dir and then putting the following in the server.properties:
>
> kafka.metrics.reporters=org.pingles.kafka.KafkaRiemannReporter
> kafka.riemann.metrics.reporter.enabled=true
> kafka.riemann.metrics.reporter.publisher=riemann
> kafka.riemann.metrics.reporter.publisher.host=127.0.0.1
> kafka.riemann.metrics.reporter.publisher.port=5555
>
> It lets you push directly from the broker to a Riemann service. However, we
> actually push messages (the Riemann events- protocol buffer messages) onto
> a Kafka topic to be later consumed by a bridged and forwarded to Riemann.
> This has the added benefit of allowing metrics to be flowed into our
> existing Redshift data load pipeline as well as being monitored in
> realtime.
>
> It's pretty early days but I deployed it to our small Kafka cluster on EC2
> earlier and have been watching the graphs all day. Although we were writing
> metrics to CSVs before we had nothing to visualise it, this makes it pretty
> easy to pump the broker metrics out.
>
> I thought I'd mention it here in case other people are in a similar
> position to us and wanted an easy (alebit still a little experimental :)
> way to get the metrics out of Kafka.
>
> I'd really appreciate some feedback, and of course I'd be delighted if
> other people use it and help fix my mistakes :)
>
> At the moment it's built against an earlier 0.8 that we're running in
> production (9bd2a) but I'm keen to start working (and deploying) an
> officially released build.
>
> Paul
>

 
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB