Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # user >> Sending huge binary files via Kafka?


Copy link to this message
-
Re: Sending huge binary files via Kafka?
The message size limit is imposed to protect the brokers and consumers from
running out of memory. The consumer does not have support for streaming a
message and has to allocate memory to be able to read the largest message.
You could try compressing the files but I'm not sure if that will get you
as much space saving to make it feasible for Kafka usage.

Thanks,
Neha
On Sep 4, 2013 5:29 AM, "Maier, Dr. Andreas" <[EMAIL PROTECTED]>
wrote:

> Hello,
>
> I have a proposal for an architecture on my desk, where people want
> to store huge binary files (like images and videos up to a size of several
> GB)
> in RiakCS. But the connection to RiakCS is supposed to work through Apache
> Kafka,
> so there will be a Kafka producer fetching the files from the source and
> sending them to a Kafka-RiakCS consumer.
> Now when I look into the Kafka configuration options
> (http://kafka.apache.org/08/configuration.html)
> I see that message.max.bytes is 1000000 by default, which would be much
> too
> small for huge binary files like videos.
> So my questions are:
> Can this size be increased to support also messages with a size
> of several GB? Has anyone already tried this? Are Kafka brokers,
> consumers and producers able to handle such a message size?
> Will setting such a huge limit on the message size have any impact
> on the performance of transporting smaller messages?
> Or should we better let our Kafka producers bypass Kafka, when
> they encounter such huge binary files at the source and
> let them store these files directly in RiakCS?
>
> Best Regards,
>
> Andreas Maier
>
>