Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # user >> Message Width


Copy link to this message
-
Re: Message Width
Each message size has a default limit of around 1 MB then, and multiple messages are batched together, presumably, up to some other default batch limit.

Has anyone needed to increase the message limit to greater than 1 MB? If so, to what limit and were there any issues?

JP
On Nov 5, 2013, at 3:05 PM, Guozhang Wang <[EMAIL PROTECTED]> wrote:

> Shafaq is right, except the default value is 1000000 bytes, so a little bit
> less than 1MB.
>
>
> On Tue, Nov 5, 2013 at 2:31 PM, Shafaq <[EMAIL PROTECTED]> wrote:
>
>> The message is sent as serialized over wire so there is no concept of width
>> but only length.
>>
>> The message size is specified in the broker config and default is 10 MB off
>> the top of my head to prevent OOM.
>> On Nov 5, 2013 1:06 PM, "Jason Pohl" <[EMAIL PROTECTED]> wrote:
>>
>>> I am new to pub-sub architecture and am looking at Kafka to be a data
>> hub.
>>>
>>> My question is there any limit to how wide a message should be in terms
>> of
>>> number of fields. Not so much an absolute limit or a theoretical one, but
>>> more of a practical limit.
>>>
>>> What is the "widest" message that you typically process?
>>>
>>> JP
>
>
>
> --
> -- Guozhang

 
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB