Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # user >> Kafka in AWS?


Copy link to this message
-
Re: Kafka in AWS?
The primary use case for Kafka is to use it on AWS...???

Sorry if I put words you didn't intend in your mouth :P ... I just thought
that sounded funny ;)

Sorry for being off-topic. Carry on :/ !

--
Felix

On Tue, Mar 20, 2012 at 6:23 PM, Russell Jurney <[EMAIL PROTECTED]>wrote:

> Yeah, that is the part I am hoping someone will contribute :)  I know I can
> write that myself.  I also know it will be buggy and that I will have lots
> of trouble.
>
> If you contribute this code, it would be a huge boon to Kafka.  It is imo
> the primary use case for Kafka atm... if only the code gets into git.
>
> On Tue, Mar 20, 2012 at 3:04 PM, Niek Sanders <[EMAIL PROTECTED]
> >wrote:
>
> > Russell,
> >
> > I'm actually in the process of writing a Java code to go from Kafka
> > messages to S3.  I might be able to rip-out my application-specific
> > parts and share something later tonight.
> >
> > The biggest hassle is that you can't append to existing S3 files.  So
> > unless you're planning on uploading each message as a separate S3
> > object, this means you need message aggregation smarts on the Kafka
> > consumer / S3 uploader side of things.
> >
> > Best,
> > Niek
> >
> >
> >
> >
> >
> >
> > On Tue, Mar 20, 2012 at 12:56 PM, Russell Jurney
> > <[EMAIL PROTECTED]> wrote:
> > > I wish someone would publish some source that writes events to S3.
> > >
> > > Russell Jurney
> > > twitter.com/rjurney
> > > [EMAIL PROTECTED]
> > > datasyndrome.com
> > >
> > > On Mar 20, 2012, at 11:20 AM, Dave Fayram <[EMAIL PROTECTED]> wrote:
> > >
> > >> We've been successfully using Kafka on AWS as well, and JMX wise we
> > >> just use an SSH tunnel.
> > >>
> > >> In general, we've been very happy with the performance on AWS, which
> > >> some people have reservations about due to the I/O situation on most
> > >> Amazon boxes.
> > >>
> > >> On Tue, Mar 20, 2012 at 9:07 AM, Gautam Singaraju
> > >> <[EMAIL PROTECTED]> wrote:
> > >>> We are have been considering Kafka for a new Data Platform. Has
> someone
> > >>> used Kafka in AWS? If so, could you please share your experiences
> with
> > us?
> > >>>
> > >>> Thank you!
> > >>> ---
> > >>> Gautam
> > >>
> > >>
> > >>
> > >> --
> > >> --
> > >> Dave Fayram
> > >> [EMAIL PROTECTED]
> >
>
>
>
> --
> Russell Jurney twitter.com/rjurney [EMAIL PROTECTED]
> datasyndrome.com
>
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB