Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Kafka >> mail # user >> Monitoring with Riemann

Paul Ingles 2013-07-21, 05:26
Copy link to this message
Re: Monitoring with Riemann

Thanks for sharing.

On Thu, Jul 18, 2013 at 2:00 PM, Paul Ingles <[EMAIL PROTECTED]> wrote:

> Hi,
> Earlier today I finished work on a library that adds an additional metrics
> reporter (https://github.com/pingles/kafka-riemann-reporter). I did it
> because although we were writing the CSV metrics we had nothing to display
> the data.
> It pushes metrics to Riemann (http://riemann.io) a very cool tool for
> aggregating and processing event streams. We've experimented with it a
> little in the past but this is the first production integration we've had.
> Streams are processed by Riemann and then plotted by Librato (
> http://librato.com).
> Installing requires building the project and copying the JAR to Kafka's
> ./libs dir and then putting the following in the server.properties:
> kafka.metrics.reporters=org.pingles.kafka.KafkaRiemannReporter
> kafka.riemann.metrics.reporter.enabled=true
> kafka.riemann.metrics.reporter.publisher=riemann
> kafka.riemann.metrics.reporter.publisher.host=
> kafka.riemann.metrics.reporter.publisher.port=5555
> It lets you push directly from the broker to a Riemann service. However, we
> actually push messages (the Riemann events- protocol buffer messages) onto
> a Kafka topic to be later consumed by a bridged and forwarded to Riemann.
> This has the added benefit of allowing metrics to be flowed into our
> existing Redshift data load pipeline as well as being monitored in
> realtime.
> It's pretty early days but I deployed it to our small Kafka cluster on EC2
> earlier and have been watching the graphs all day. Although we were writing
> metrics to CSVs before we had nothing to visualise it, this makes it pretty
> easy to pump the broker metrics out.
> I thought I'd mention it here in case other people are in a similar
> position to us and wanted an easy (alebit still a little experimental :)
> way to get the metrics out of Kafka.
> I'd really appreciate some feedback, and of course I'd be delighted if
> other people use it and help fix my mistakes :)
> At the moment it's built against an earlier 0.8 that we're running in
> production (9bd2a) but I'm keen to start working (and deploying) an
> officially released build.
> Paul