Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka, mail # user - S3 Archiving for Kafka topics (with Zookeeper resume)


Copy link to this message
-
Re: S3 Archiving for Kafka topics (with Zookeeper resume)
S Ahmed 2013-01-31, 14:58
Great thanks.

BTW, that's not a daemon server is it?  Is it something you have to wrap to
run as a service/daemon?
On Thu, Jan 31, 2013 at 4:29 AM, Jibran Saithi <[EMAIL PROTECTED]>wrote:

> Hey,
>
> I know this has come up a few times, so thought I'd share a bit of code
> we've been using to archive topics to S3.
>
> Particularly unimaginatively named, but is available here:
> https://github.com/jibs/kafka-s3-consumer
>
> We needed something with Zookeeper support for storing the offsets, but
> didn't come across anything so I quickly put this together. For the moment
> I've removed graphite stats reporting because it has a few internal
> dependencies, but plan to sort that out soon.
>
> Hope this helps,
> Jibran
>