Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka, mail # user - Message Width


Copy link to this message
-
Re: Message Width
Jason Pohl 2013-11-06, 21:03
Each message size has a default limit of around 1 MB then, and multiple messages are batched together, presumably, up to some other default batch limit.

Has anyone needed to increase the message limit to greater than 1 MB? If so, to what limit and were there any issues?

JP
On Nov 5, 2013, at 3:05 PM, Guozhang Wang <[EMAIL PROTECTED]> wrote:

> Shafaq is right, except the default value is 1000000 bytes, so a little bit
> less than 1MB.
>
>
> On Tue, Nov 5, 2013 at 2:31 PM, Shafaq <[EMAIL PROTECTED]> wrote:
>
>> The message is sent as serialized over wire so there is no concept of width
>> but only length.
>>
>> The message size is specified in the broker config and default is 10 MB off
>> the top of my head to prevent OOM.
>> On Nov 5, 2013 1:06 PM, "Jason Pohl" <[EMAIL PROTECTED]> wrote:
>>
>>> I am new to pub-sub architecture and am looking at Kafka to be a data
>> hub.
>>>
>>> My question is there any limit to how wide a message should be in terms
>> of
>>> number of fields. Not so much an absolute limit or a theoretical one, but
>>> more of a practical limit.
>>>
>>> What is the "widest" message that you typically process?
>>>
>>> JP
>
>
>
> --
> -- Guozhang