Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # user >> Kafka in AWS?

Copy link to this message
Re: Kafka in AWS?
I would use the payload. I want the message to be exactly as it is. We want
to name the files as per topic.
(That's how we differentiate right now).


On Wed, Mar 21, 2012 at 11:01 AM, Niek Sanders <[EMAIL PROTECTED]>wrote:

> So what would you like the S3 files to actually look like?
> One Kafka message body per line?  Should the message topic be tossed
> in there too?
> A tricky aspect is that the Kafka message body is an opaque byte
> array.  For my own case I'm using JSON for the payload so it makes my
> requirements simpler.
> - Niek
> On Tue, Mar 20, 2012 at 10:07 PM, Russell Jurney
> <[EMAIL PROTECTED]> wrote:
> > I want events in S3 to process them in Hadoop. I'd like to emit them in
> my app, and have them magically show up in 64MB chunks on S3. Like most
> everyone else.
> >
> > Russell Jurney http://datasyndrome.com
> >