Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Kafka >> mail # user >> any message size restrictions? suggest range?


+
S Ahmed 2011-12-21, 20:15
+
Neha Narkhede 2011-12-21, 21:16
Copy link to this message
-
Re: any message size restrictions? suggest range?
What would be an upper bound then? i.e. 100K should be ok, what shouldn't?
:)

On Wed, Dec 21, 2011 at 4:16 PM, Neha Narkhede <[EMAIL PROTECTED]>wrote:

> >> Was kafka designed for a specific message size range?
>
> Kafka consumer reads a message from the socket into memory. If a
> message is large enough to cause OutOfMemoryException, then the Kafka
> consumer is unable to return more messages from the socket byte
> buffer. This can be fixed by enabling the Kafka consumers to have a
> 'streaming' API, where such large messages could be read in a
> piecemeal fashion. But it is tricky and we don't have that feature
> yet.
>
> To avoid your Kafka consumer from getting into a bad state due to a
> large message, you can set "max.message.size" to the largest possible
> message size on your producer. Any message larger than that never
> enters the Kafka cluster and hence never reaches a Kafka consumer.
>
> >> > Seeing as it is used to aggregate log messages, is it safe to say
> message
> > sizes of 2-100K are reasonable and won't cause any issues?
>
> 100K message sizes should work fine.
>
> Thanks,
> Neha
>
> On Wed, Dec 21, 2011 at 12:15 PM, S Ahmed <[EMAIL PROTECTED]> wrote:
> > Was kafka designed for a specific message size range?
> >
> > Seeing as it is used to aggregate log messages, is it safe to say message
> > sizes of 2-100K are reasonable and won't cause any issues?
>
+
Neha Narkhede 2011-12-21, 21:27
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB