Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Kafka >> mail # user >> S3 Archiving for Kafka topics (with Zookeeper resume)


+
Jibran Saithi 2013-01-31, 09:30
+
S Ahmed 2013-01-31, 14:58
+
Jibran Saithi 2013-01-31, 17:39
Copy link to this message
-
Re: S3 Archiving for Kafka topics (with Zookeeper resume)
Cool, well definitely add it to that ecosystem page to help people find it.

-Jay

On Thu, Jan 31, 2013 at 9:38 AM, Jibran Saithi <[EMAIL PROTECTED]>wrote:

> Yeah, we use supervisor which seems to work fine for us.
>
> On 31 January 2013 14:57, S Ahmed <[EMAIL PROTECTED]> wrote:
>
> > Great thanks.
> >
> > BTW, that's not a daemon server is it?  Is it something you have to wrap
> to
> > run as a service/daemon?
> >
> >
> > On Thu, Jan 31, 2013 at 4:29 AM, Jibran Saithi <[EMAIL PROTECTED]
> > >wrote:
> >
> > > Hey,
> > >
> > > I know this has come up a few times, so thought I'd share a bit of code
> > > we've been using to archive topics to S3.
> > >
> > > Particularly unimaginatively named, but is available here:
> > > https://github.com/jibs/kafka-s3-consumer
> > >
> > > We needed something with Zookeeper support for storing the offsets, but
> > > didn't come across anything so I quickly put this together. For the
> > moment
> > > I've removed graphite stats reporting because it has a few internal
> > > dependencies, but plan to sort that out soon.
> > >
> > > Hope this helps,
> > > Jibran
> > >
> >
>
>
>
> --
> Jibran Saithi
> Product Engineer
>
>
> QuBit Digital Ltd
> www.qubitdigital.com
> *
> *
> Office: +44 (0)203 411 9130
>
> QuBit Digital Ltd
> 20 Broadwick Street
> London
> W1F 8HT
>
>
>
> This email may be confidential or privileged. If you received this
> communication by mistake, please don't forward it to anyone else, please
> erase all copies and attachments, and please let me know that it has gone
> to
> the wrong person. Thanks
>

 
+
San 2013-01-31, 17:55
+
Jay Kreps 2013-01-31, 17:05
+
Jibran Saithi 2013-01-31, 17:38