Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Kafka >> mail # user >> S3 Archiving for Kafka topics (with Zookeeper resume)


+
Jibran Saithi 2013-01-31, 09:30
+
S Ahmed 2013-01-31, 14:58
+
Jibran Saithi 2013-01-31, 17:39
Copy link to this message
-
Re: S3 Archiving for Kafka topics (with Zookeeper resume)
Cool, well definitely add it to that ecosystem page to help people find it.

-Jay

On Thu, Jan 31, 2013 at 9:38 AM, Jibran Saithi <[EMAIL PROTECTED]>wrote:

> Yeah, we use supervisor which seems to work fine for us.
>
> On 31 January 2013 14:57, S Ahmed <[EMAIL PROTECTED]> wrote:
>
> > Great thanks.
> >
> > BTW, that's not a daemon server is it?  Is it something you have to wrap
> to
> > run as a service/daemon?
> >
> >
> > On Thu, Jan 31, 2013 at 4:29 AM, Jibran Saithi <[EMAIL PROTECTED]
> > >wrote:
> >
> > > Hey,
> > >
> > > I know this has come up a few times, so thought I'd share a bit of code
> > > we've been using to archive topics to S3.
> > >
> > > Particularly unimaginatively named, but is available here:
> > > https://github.com/jibs/kafka-s3-consumer
> > >
> > > We needed something with Zookeeper support for storing the offsets, but
> > > didn't come across anything so I quickly put this together. For the
> > moment
> > > I've removed graphite stats reporting because it has a few internal
> > > dependencies, but plan to sort that out soon.
> > >
> > > Hope this helps,
> > > Jibran
> > >
> >
>
>
>
> --
> Jibran Saithi
> Product Engineer
>
>
> QuBit Digital Ltd
> www.qubitdigital.com
> *
> *
> Office: +44 (0)203 411 9130
>
> QuBit Digital Ltd
> 20 Broadwick Street
> London
> W1F 8HT
>
>
>
> This email may be confidential or privileged. If you received this
> communication by mistake, please don't forward it to anyone else, please
> erase all copies and attachments, and please let me know that it has gone
> to
> the wrong person. Thanks
>

 
+
San 2013-01-31, 17:55
+
Jay Kreps 2013-01-31, 17:05
+
Jibran Saithi 2013-01-31, 17:38
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB