Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Kafka >> mail # user >> S3 Archiving for Kafka topics (with Zookeeper resume)


+
Jibran Saithi 2013-01-31, 09:30
Copy link to this message
-
Re: S3 Archiving for Kafka topics (with Zookeeper resume)
Great thanks.

BTW, that's not a daemon server is it?  Is it something you have to wrap to
run as a service/daemon?
On Thu, Jan 31, 2013 at 4:29 AM, Jibran Saithi <[EMAIL PROTECTED]>wrote:

> Hey,
>
> I know this has come up a few times, so thought I'd share a bit of code
> we've been using to archive topics to S3.
>
> Particularly unimaginatively named, but is available here:
> https://github.com/jibs/kafka-s3-consumer
>
> We needed something with Zookeeper support for storing the offsets, but
> didn't come across anything so I quickly put this together. For the moment
> I've removed graphite stats reporting because it has a few internal
> dependencies, but plan to sort that out soon.
>
> Hope this helps,
> Jibran
>

 
+
Jibran Saithi 2013-01-31, 17:39
+
Jay Kreps 2013-01-31, 17:41
+
San 2013-01-31, 17:55
+
Jay Kreps 2013-01-31, 17:05
+
Jibran Saithi 2013-01-31, 17:38
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB