Search Hadoop and all its sub project:

Switch to Threaded View
Subject: Re: Question on message content, compression, multiple messages per kafka message?
Using a single Kafka message to contain an application snapshot has the
upside of getting atomicity for free. Either the snapshot will be written
as a whole to Kafka or not. This is poor man's transactionality. Care needs
to be taken to ensure that the message is not too large since that might
cause memory consumption problems on the server or the consumers.

As far as compression overhead is concerned, have you tried running Snappy?
Snappy's performance is good enough to offset the decompression-compression
overhead on the server.

On Thu, Jun 26, 2014 at 12:42 PM, Bert Corderman <[EMAIL PROTECTED]> wrote:
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB