Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Flume >> mail # user >> Max Event Size


Hi Kevin,

The eventsize vs. capacity slot size is defined in "flume-ng-core/src/main/java/org/apache/flume/channel/MemoryChannel.java" via:

int eventByteSize = (int)Math.ceil(estimateEventSize(event)/byteCapacitySlotSize);

and will later throw out a error:
throw new ChannelException("Put queue for MemoryTransaction of capacity " +
              putList.size() + " full, consider committing more frequently, " +
              "increasing capacity or increasing thread count");

Can you use FileChannel instead of MemoryChannel, just for testing?
http://flume.apache.org/FlumeUserGuide.html#file-channel

Cheers,
 Alex

On Nov 24, 2012, at 10:33 AM, Lichen <[EMAIL PROTECTED]> wrote:

> Hi Alex,
>
> I don't see any errors/warning with DEBUG.
>
> Here is the problem: if I tried to read a line larger than 2.5 MB and send via memory channel, the sink lost it and won't receive anything afterwards.
>
> Here is the configuration I tried:
>
> Source: exec. I tried both cat/tail command, and a Java program to read lines from file and print to stdout.
> Channel: memory channel and file channel. I think capacity of memory channel is about the size of the queue of events, which won't affect the max size of an event.
> Sink: both HDFS and Fill-Roll. They are able to receive short events before large events come.
>
> Can you suggest what to configure? I'm using Flume 1.2.
>
> Thanks,
>
> Kevin
>
> On Nov 23, 2012, at 6:34 PM, Alexander Alten-Lorenz <[EMAIL PROTECTED]> wrote:
>
> Kevin,
>
> This depends on your memory configuration. When you start flume with DEBUG did you got some errors / warning?
>
> Cheers,
> Alex
>
> P.S. http://flume.apache.org/FlumeUserGuide.html#exec-source
>
>
> On Nov 23, 2012, at 10:29 AM, Lichen <[EMAIL PROTECTED]> wrote:
>
>> Hi all,
>>
>> I tried to use exec source to tail a file of which each line is few MB. The File-Roll/HDFS sinks received nothing in this case, while work fine with other short lines. Is it because of the maximum size of a single event? What parameter to set if it's configurable.
>>
>> Thanks
>>
>> Kevin
>
> --
> Alexander Alten-Lorenz
> http://mapredit.blogspot.com
> German Hadoop LinkedIn Group: http://goo.gl/N8pCF
>

--
Alexander Alten-Lorenz
http://mapredit.blogspot.com
German Hadoop LinkedIn Group: http://goo.gl/N8pCF
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB