Paul Ingles 2013-07-21, 05:26
-Re: Monitoring with Riemann
Jun Rao 2013-07-22, 05:37
Thanks for sharing.
On Thu, Jul 18, 2013 at 2:00 PM, Paul Ingles <[EMAIL PROTECTED]> wrote:
> Earlier today I finished work on a library that adds an additional metrics
> reporter (https://github.com/pingles/kafka-riemann-reporter). I did it
> because although we were writing the CSV metrics we had nothing to display
> the data.
> It pushes metrics to Riemann (http://riemann.io) a very cool tool for
> aggregating and processing event streams. We've experimented with it a
> little in the past but this is the first production integration we've had.
> Streams are processed by Riemann and then plotted by Librato (
> Installing requires building the project and copying the JAR to Kafka's
> ./libs dir and then putting the following in the server.properties:
> It lets you push directly from the broker to a Riemann service. However, we
> actually push messages (the Riemann events- protocol buffer messages) onto
> a Kafka topic to be later consumed by a bridged and forwarded to Riemann.
> This has the added benefit of allowing metrics to be flowed into our
> existing Redshift data load pipeline as well as being monitored in
> It's pretty early days but I deployed it to our small Kafka cluster on EC2
> earlier and have been watching the graphs all day. Although we were writing
> metrics to CSVs before we had nothing to visualise it, this makes it pretty
> easy to pump the broker metrics out.
> I thought I'd mention it here in case other people are in a similar
> position to us and wanted an easy (alebit still a little experimental :)
> way to get the metrics out of Kafka.
> I'd really appreciate some feedback, and of course I'd be delighted if
> other people use it and help fix my mistakes :)
> At the moment it's built against an earlier 0.8 that we're running in
> production (9bd2a) but I'm keen to start working (and deploying) an
> officially released build.