Kafka, mail # user - Re: Sending huge binary files via Kafka? - 2013-09-06, 08:07
 Search Hadoop and all its subprojects:

Switch to Plain View
+
Maier, Dr. Andreas 2013-09-04, 12:29
+
Neha Narkhede 2013-09-04, 14:26
+
Magnus Edenhill 2013-09-05, 10:29
Copy link to this message
-
Re: Sending huge binary files via Kafka?
Magnus,

this sounds like an interesting idea. But at the actual state of Kafka
this would mean, that I would have to extend the Kafka Producer and
Consumer
classes on my own to support that kind of message/file transfer, wouldn't
it?
This is a bit too much effort for me at the moment.
But of course it would be nice, if Kafka Producers or Consumers would
support
zero-copy file transfer natively.

At the moment I'm more thinking about sending a message to the consumer
with an URL of the huge binary file, and let the consumer fetch the file
from that
URL directly. By that we would use Kafka only for sending a notification,
that
a new file exists at the source. The real file transfer would bypass
the Kafka queue.

Andreas Maier

Am 05.09.13 12:28 schrieb "Magnus Edenhill" unter <[EMAIL PROTECTED]>:

 
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB