Hey,

I know this has come up a few times, so thought I'd share a bit of code
we've been using to archive topics to S3.

Particularly unimaginatively named, but is available here:
https://github.com/jibs/kafka-s3-consumer

We needed something with Zookeeper support for storing the offsets, but
didn't come across anything so I quickly put this together. For the moment
I've removed graphite stats reporting because it has a few internal
dependencies, but plan to sort that out soon.

Hope this helps,
Jibran

 

Great thanks.

BTW, that's not a daemon server is it?  Is it something you have to wrap to
run as a service/daemon?
On Thu, Jan 31, 2013 at 4:29 AM, Jibran Saithi <[EMAIL PROTECTED]>wrote:

> Hey,
>
> I know this has come up a few times, so thought I'd share a bit of code
> we've been using to archive topics to S3.
>
> Particularly unimaginatively named, but is available here:
> https://github.com/jibs/kafka-s3-consumer
>
> We needed something with Zookeeper support for storing the offsets, but
> didn't come across anything so I quickly put this together. For the moment
> I've removed graphite stats reporting because it has a few internal
> dependencies, but plan to sort that out soon.
>
> Hope this helps,
> Jibran
>

 

This is super cool. Are we linking the right repo on the ecosystem page?

https://cwiki.apache.org/confluence/display/KAFKA/Ecosystem

-Jay
On Thu, Jan 31, 2013 at 1:29 AM, Jibran Saithi <[EMAIL PROTECTED]>wrote:

> Hey,
>
> I know this has come up a few times, so thought I'd share a bit of code
> we've been using to archive topics to S3.
>
> Particularly unimaginatively named, but is available here:
> https://github.com/jibs/kafka-s3-consumer
>
> We needed something with Zookeeper support for storing the offsets, but
> didn't come across anything so I quickly put this together. For the moment
> I've removed graphite stats reporting because it has a few internal
> dependencies, but plan to sort that out soon.
>
> Hope this helps,
> Jibran
>

 

I should probably call it something else to avoid confusion, it started as
a fork but since became a re-write (the linked one was using simpleconsumer
and we needed some Zookeeper goodness).

On 31 January 2013 17:04, Jay Kreps <[EMAIL PROTECTED]> wrote:

> This is super cool. Are we linking the right repo on the ecosystem page?
>
> https://cwiki.apache.org/confluence/display/KAFKA/Ecosystem
>
> -Jay
>
>
> On Thu, Jan 31, 2013 at 1:29 AM, Jibran Saithi <[EMAIL PROTECTED]
> >wrote:
>
> > Hey,
> >
> > I know this has come up a few times, so thought I'd share a bit of code
> > we've been using to archive topics to S3.
> >
> > Particularly unimaginatively named, but is available here:
> > https://github.com/jibs/kafka-s3-consumer
> >
> > We needed something with Zookeeper support for storing the offsets, but
> > didn't come across anything so I quickly put this together. For the
> moment
> > I've removed graphite stats reporting because it has a few internal
> > dependencies, but plan to sort that out soon.
> >
> > Hope this helps,
> > Jibran
> >
>

--
Jibran Saithi
Product Engineer
QuBit Digital Ltd
www.qubitdigital.com
*
*
Office: +44 (0)203 411 9130

QuBit Digital Ltd
20 Broadwick Street
London
W1F 8HT

This email may be confidential or privileged. If you received this
communication by mistake, please don't forward it to anyone else, please
erase all copies and attachments, and please let me know that it has gone to
the wrong person. Thanks

 

Yeah, we use supervisor which seems to work fine for us.

On 31 January 2013 14:57, S Ahmed <[EMAIL PROTECTED]> wrote:

> Great thanks.
>
> BTW, that's not a daemon server is it?  Is it something you have to wrap to
> run as a service/daemon?
>
>
> On Thu, Jan 31, 2013 at 4:29 AM, Jibran Saithi <[EMAIL PROTECTED]
> >wrote:
>
> > Hey,
> >
> > I know this has come up a few times, so thought I'd share a bit of code
> > we've been using to archive topics to S3.
> >
> > Particularly unimaginatively named, but is available here:
> > https://github.com/jibs/kafka-s3-consumer
> >
> > We needed something with Zookeeper support for storing the offsets, but
> > didn't come across anything so I quickly put this together. For the
> moment
> > I've removed graphite stats reporting because it has a few internal
> > dependencies, but plan to sort that out soon.
> >
> > Hope this helps,
> > Jibran
> >
>

--
Jibran Saithi
Product Engineer
QuBit Digital Ltd
www.qubitdigital.com
*
*
Office: +44 (0)203 411 9130

QuBit Digital Ltd
20 Broadwick Street
London
W1F 8HT

This email may be confidential or privileged. If you received this
communication by mistake, please don't forward it to anyone else, please
erase all copies and attachments, and please let me know that it has gone to
the wrong person. Thanks

 

Cool, well definitely add it to that ecosystem page to help people find it.

-Jay

On Thu, Jan 31, 2013 at 9:38 AM, Jibran Saithi <[EMAIL PROTECTED]>wrote:

> Yeah, we use supervisor which seems to work fine for us.
>
> On 31 January 2013 14:57, S Ahmed <[EMAIL PROTECTED]> wrote:
>
> > Great thanks.
> >
> > BTW, that's not a daemon server is it?  Is it something you have to wrap
> to
> > run as a service/daemon?
> >
> >
> > On Thu, Jan 31, 2013 at 4:29 AM, Jibran Saithi <[EMAIL PROTECTED]
> > >wrote:
> >
> > > Hey,
> > >
> > > I know this has come up a few times, so thought I'd share a bit of code
> > > we've been using to archive topics to S3.
> > >
> > > Particularly unimaginatively named, but is available here:
> > > https://github.com/jibs/kafka-s3-consumer
> > >
> > > We needed something with Zookeeper support for storing the offsets, but
> > > didn't come across anything so I quickly put this together. For the
> > moment
> > > I've removed graphite stats reporting because it has a few internal
> > > dependencies, but plan to sort that out soon.
> > >
> > > Hope this helps,
> > > Jibran
> > >
> >
>
>
>
> --
> Jibran Saithi
> Product Engineer
>
>
> QuBit Digital Ltd
> www.qubitdigital.com
> *
> *
> Office: +44 (0)203 411 9130
>
> QuBit Digital Ltd
> 20 Broadwick Street
> London
> W1F 8HT
>
>
>
> This email may be confidential or privileged. If you received this
> communication by mistake, please don't forward it to anyone else, please
> erase all copies and attachments, and please let me know that it has gone
> to
> the wrong person. Thanks
>

 

I see the main method, once you run this will it stop execution or it will keep waiting and processing is what I mean?

Supervisord from what I understand is used to ensure the process stats up after reboots and restarts if it crashes etc

Sent from my iPhone

On 2013-01-31, at 12:38 PM, Jibran Saithi <[EMAIL PROTECTED]> wrote:

> Yeah, we use supervisor which seems to work fine for us.
>
> On 31 January 2013 14:57, S Ahmed <[EMAIL PROTECTED]> wrote:
>
>> Great thanks.
>>
>> BTW, that's not a daemon server is it?  Is it something you have to wrap to
>> run as a service/daemon?
>>
>>
>> On Thu, Jan 31, 2013 at 4:29 AM, Jibran Saithi <[EMAIL PROTECTED]
>>> wrote:
>>
>>> Hey,
>>>
>>> I know this has come up a few times, so thought I'd share a bit of code
>>> we've been using to archive topics to S3.
>>>
>>> Particularly unimaginatively named, but is available here:
>>> https://github.com/jibs/kafka-s3-consumer
>>>
>>> We needed something with Zookeeper support for storing the offsets, but
>>> didn't come across anything so I quickly put this together. For the
>> moment
>>> I've removed graphite stats reporting because it has a few internal
>>> dependencies, but plan to sort that out soon.
>>>
>>> Hope this helps,
>>> Jibran
>
>
>
> --
> Jibran Saithi
> Product Engineer
>
>
> QuBit Digital Ltd
> www.qubitdigital.com
> *
> *
> Office: +44 (0)203 411 9130
>
> QuBit Digital Ltd
> 20 Broadwick Street
> London
> W1F 8HT
>
>
>
> This email may be confidential or privileged. If you received this
> communication by mistake, please don't forward it to anyone else, please
> erase all copies and attachments, and please let me know that it has gone to
> the wrong person. Thanks