Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Kafka >> mail # user >> Payload size exception


+
Bo Sun 2013-01-29, 08:18
+
Jay Kreps 2013-01-29, 16:05
+
Neha Narkhede 2013-01-29, 16:35
+
S Ahmed 2013-01-29, 16:57
+
Xavier Stevens 2013-01-29, 17:21
+
Neha Narkhede 2013-01-29, 17:28
+
S Ahmed 2013-01-29, 19:00
Copy link to this message
-
Re: Payload size exception
Ack, right you are Neha, my bad.

WRT to how to set the maximum there are two considerations:
1. It should be smaller then the fetch size your consumers use
2. Messages are fully instantiated in memory so obscenely large messages
(say hundreds of mb) will cause a lot of memory allocation churn/problems.

-Jay
On Tue, Jan 29, 2013 at 8:57 AM, S Ahmed <[EMAIL PROTECTED]> wrote:

> Neha/Jay,
>
> At linkedin, what is the largest payload size per message you guys have in
> production?  My app might have like 20-100 kilobytes in size and I am
> hoping to get an idea if others have large messages like this for any
> production use case.
>
>
> On Tue, Jan 29, 2013 at 11:35 AM, Neha Narkhede <[EMAIL PROTECTED]
> >wrote:
>
> > > In 0.7.x this
> > > setting is controlled by the broker configuration max.message.size.
> > >
> >
> > Actually, in 0.7.x this setting is controlled by max.message.size on the
> > producer. In 0.8, we moved this setting to the broker.
> >
> > Thanks,
> > Neha
> >
>

 
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB