Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Kafka >> mail # user >> S3 Archiving for Kafka topics (with Zookeeper resume)


+
Jibran Saithi 2013-01-31, 09:30
+
S Ahmed 2013-01-31, 14:58
+
Jibran Saithi 2013-01-31, 17:39
+
Jay Kreps 2013-01-31, 17:41
Copy link to this message
-
Re: S3 Archiving for Kafka topics (with Zookeeper resume)
I see the main method, once you run this will it stop execution or it will keep waiting and processing is what I mean?

Supervisord from what I understand is used to ensure the process stats up after reboots and restarts if it crashes etc

Sent from my iPhone

On 2013-01-31, at 12:38 PM, Jibran Saithi <[EMAIL PROTECTED]> wrote:

> Yeah, we use supervisor which seems to work fine for us.
>
> On 31 January 2013 14:57, S Ahmed <[EMAIL PROTECTED]> wrote:
>
>> Great thanks.
>>
>> BTW, that's not a daemon server is it?  Is it something you have to wrap to
>> run as a service/daemon?
>>
>>
>> On Thu, Jan 31, 2013 at 4:29 AM, Jibran Saithi <[EMAIL PROTECTED]
>>> wrote:
>>
>>> Hey,
>>>
>>> I know this has come up a few times, so thought I'd share a bit of code
>>> we've been using to archive topics to S3.
>>>
>>> Particularly unimaginatively named, but is available here:
>>> https://github.com/jibs/kafka-s3-consumer
>>>
>>> We needed something with Zookeeper support for storing the offsets, but
>>> didn't come across anything so I quickly put this together. For the
>> moment
>>> I've removed graphite stats reporting because it has a few internal
>>> dependencies, but plan to sort that out soon.
>>>
>>> Hope this helps,
>>> Jibran
>
>
>
> --
> Jibran Saithi
> Product Engineer
>
>
> QuBit Digital Ltd
> www.qubitdigital.com
> *
> *
> Office: +44 (0)203 411 9130
>
> QuBit Digital Ltd
> 20 Broadwick Street
> London
> W1F 8HT
>
>
>
> This email may be confidential or privileged. If you received this
> communication by mistake, please don't forward it to anyone else, please
> erase all copies and attachments, and please let me know that it has gone to
> the wrong person. Thanks

 
+
Jay Kreps 2013-01-31, 17:05
+
Jibran Saithi 2013-01-31, 17:38