Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Kafka, mail # user - S3 Archiving for Kafka topics (with Zookeeper resume)


Copy link to this message
-
S3 Archiving for Kafka topics (with Zookeeper resume)
Jibran Saithi 2013-01-31, 09:30
Hey,

I know this has come up a few times, so thought I'd share a bit of code
we've been using to archive topics to S3.

Particularly unimaginatively named, but is available here:
https://github.com/jibs/kafka-s3-consumer

We needed something with Zookeeper support for storing the offsets, but
didn't come across anything so I quickly put this together. For the moment
I've removed graphite stats reporting because it has a few internal
dependencies, but plan to sort that out soon.

Hope this helps,
Jibran

 
+
S Ahmed 2013-01-31, 14:58
+
Jibran Saithi 2013-01-31, 17:39
+
Jay Kreps 2013-01-31, 17:41
+
San 2013-01-31, 17:55
+
Jay Kreps 2013-01-31, 17:05
+
Jibran Saithi 2013-01-31, 17:38