Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # user >> any message size restrictions? suggest range?


Copy link to this message
-
Re: any message size restrictions? suggest range?
What would be an upper bound then? i.e. 100K should be ok, what shouldn't?
:)

On Wed, Dec 21, 2011 at 4:16 PM, Neha Narkhede <[EMAIL PROTECTED]>wrote:

> >> Was kafka designed for a specific message size range?
>
> Kafka consumer reads a message from the socket into memory. If a
> message is large enough to cause OutOfMemoryException, then the Kafka
> consumer is unable to return more messages from the socket byte
> buffer. This can be fixed by enabling the Kafka consumers to have a
> 'streaming' API, where such large messages could be read in a
> piecemeal fashion. But it is tricky and we don't have that feature
> yet.
>
> To avoid your Kafka consumer from getting into a bad state due to a
> large message, you can set "max.message.size" to the largest possible
> message size on your producer. Any message larger than that never
> enters the Kafka cluster and hence never reaches a Kafka consumer.
>
> >> > Seeing as it is used to aggregate log messages, is it safe to say
> message
> > sizes of 2-100K are reasonable and won't cause any issues?
>
> 100K message sizes should work fine.
>
> Thanks,
> Neha
>
> On Wed, Dec 21, 2011 at 12:15 PM, S Ahmed <[EMAIL PROTECTED]> wrote:
> > Was kafka designed for a specific message size range?
> >
> > Seeing as it is used to aggregate log messages, is it safe to say message
> > sizes of 2-100K are reasonable and won't cause any issues?
>