Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # user >> Kafka in AWS?


Copy link to this message
-
Re: Kafka in AWS?
The primary use case for Kafka is to use it on AWS...???

Sorry if I put words you didn't intend in your mouth :P ... I just thought
that sounded funny ;)

Sorry for being off-topic. Carry on :/ !

--
Felix

On Tue, Mar 20, 2012 at 6:23 PM, Russell Jurney <[EMAIL PROTECTED]>wrote:

> Yeah, that is the part I am hoping someone will contribute :)  I know I can
> write that myself.  I also know it will be buggy and that I will have lots
> of trouble.
>
> If you contribute this code, it would be a huge boon to Kafka.  It is imo
> the primary use case for Kafka atm... if only the code gets into git.
>
> On Tue, Mar 20, 2012 at 3:04 PM, Niek Sanders <[EMAIL PROTECTED]
> >wrote:
>
> > Russell,
> >
> > I'm actually in the process of writing a Java code to go from Kafka
> > messages to S3.  I might be able to rip-out my application-specific
> > parts and share something later tonight.
> >
> > The biggest hassle is that you can't append to existing S3 files.  So
> > unless you're planning on uploading each message as a separate S3
> > object, this means you need message aggregation smarts on the Kafka
> > consumer / S3 uploader side of things.
> >
> > Best,
> > Niek
> >
> >
> >
> >
> >
> >
> > On Tue, Mar 20, 2012 at 12:56 PM, Russell Jurney
> > <[EMAIL PROTECTED]> wrote:
> > > I wish someone would publish some source that writes events to S3.
> > >
> > > Russell Jurney
> > > twitter.com/rjurney
> > > [EMAIL PROTECTED]
> > > datasyndrome.com
> > >
> > > On Mar 20, 2012, at 11:20 AM, Dave Fayram <[EMAIL PROTECTED]> wrote:
> > >
> > >> We've been successfully using Kafka on AWS as well, and JMX wise we
> > >> just use an SSH tunnel.
> > >>
> > >> In general, we've been very happy with the performance on AWS, which
> > >> some people have reservations about due to the I/O situation on most
> > >> Amazon boxes.
> > >>
> > >> On Tue, Mar 20, 2012 at 9:07 AM, Gautam Singaraju
> > >> <[EMAIL PROTECTED]> wrote:
> > >>> We are have been considering Kafka for a new Data Platform. Has
> someone
> > >>> used Kafka in AWS? If so, could you please share your experiences
> with
> > us?
> > >>>
> > >>> Thank you!
> > >>> ---
> > >>> Gautam
> > >>
> > >>
> > >>
> > >> --
> > >> --
> > >> Dave Fayram
> > >> [EMAIL PROTECTED]
> >
>
>
>
> --
> Russell Jurney twitter.com/rjurney [EMAIL PROTECTED]
> datasyndrome.com
>