Kafka, mail # user - Re: Experiences with larger message sizes - 2014-06-24, 16:26
 Search Hadoop and all its subprojects:

Switch to Threaded View
Copy link to this message
Re: Experiences with larger message sizes
Hi Denny, have you considered saving those files to HDFS and sending the
"event" information to Kafka?

You could then pass that off to Apache Spark in a consumer and get data
locality for the file saved (or something of the sort [no pun intended]).

You could also stream every line (or however you want to "chunk" it) in the
file as a separate message to the broker with a wrapping message object (so
you know which file you are dealing with when consuming).

What you plan to-do with the data has a lot to-do with how you are going to
process and manage it.

 Joe Stein
 Founder, Principal Consultant
 Big Data Open Source Security LLC
 Twitter: @allthingshadoop <http://www.twitter.com/allthingshadoop>
On Tue, Jun 24, 2014 at 11:35 AM, Denny Lee <[EMAIL PROTECTED]> wrote:
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB