Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # user >> Kafka in AWS?


Copy link to this message
-
Re: Kafka in AWS?
Yeah, that is the part I am hoping someone will contribute :)  I know I can
write that myself.  I also know it will be buggy and that I will have lots
of trouble.

If you contribute this code, it would be a huge boon to Kafka.  It is imo
the primary use case for Kafka atm... if only the code gets into git.

On Tue, Mar 20, 2012 at 3:04 PM, Niek Sanders <[EMAIL PROTECTED]>wrote:

> Russell,
>
> I'm actually in the process of writing a Java code to go from Kafka
> messages to S3.  I might be able to rip-out my application-specific
> parts and share something later tonight.
>
> The biggest hassle is that you can't append to existing S3 files.  So
> unless you're planning on uploading each message as a separate S3
> object, this means you need message aggregation smarts on the Kafka
> consumer / S3 uploader side of things.
>
> Best,
> Niek
>
>
>
>
>
>
> On Tue, Mar 20, 2012 at 12:56 PM, Russell Jurney
> <[EMAIL PROTECTED]> wrote:
> > I wish someone would publish some source that writes events to S3.
> >
> > Russell Jurney
> > twitter.com/rjurney
> > [EMAIL PROTECTED]
> > datasyndrome.com
> >
> > On Mar 20, 2012, at 11:20 AM, Dave Fayram <[EMAIL PROTECTED]> wrote:
> >
> >> We've been successfully using Kafka on AWS as well, and JMX wise we
> >> just use an SSH tunnel.
> >>
> >> In general, we've been very happy with the performance on AWS, which
> >> some people have reservations about due to the I/O situation on most
> >> Amazon boxes.
> >>
> >> On Tue, Mar 20, 2012 at 9:07 AM, Gautam Singaraju
> >> <[EMAIL PROTECTED]> wrote:
> >>> We are have been considering Kafka for a new Data Platform. Has someone
> >>> used Kafka in AWS? If so, could you please share your experiences with
> us?
> >>>
> >>> Thank you!
> >>> ---
> >>> Gautam
> >>
> >>
> >>
> >> --
> >> --
> >> Dave Fayram
> >> [EMAIL PROTECTED]
>

--
Russell Jurney twitter.com/rjurney [EMAIL PROTECTED] datasyndrome.com