Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka, mail # user - Payload size exception


Copy link to this message
-
Re: Payload size exception
Jay Kreps 2013-01-29, 18:44
Ack, right you are Neha, my bad.

WRT to how to set the maximum there are two considerations:
1. It should be smaller then the fetch size your consumers use
2. Messages are fully instantiated in memory so obscenely large messages
(say hundreds of mb) will cause a lot of memory allocation churn/problems.

-Jay
On Tue, Jan 29, 2013 at 8:57 AM, S Ahmed <[EMAIL PROTECTED]> wrote:

> Neha/Jay,
>
> At linkedin, what is the largest payload size per message you guys have in
> production?  My app might have like 20-100 kilobytes in size and I am
> hoping to get an idea if others have large messages like this for any
> production use case.
>
>
> On Tue, Jan 29, 2013 at 11:35 AM, Neha Narkhede <[EMAIL PROTECTED]
> >wrote:
>
> > > In 0.7.x this
> > > setting is controlled by the broker configuration max.message.size.
> > >
> >
> > Actually, in 0.7.x this setting is controlled by max.message.size on the
> > producer. In 0.8, we moved this setting to the broker.
> >
> > Thanks,
> > Neha
> >
>