Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Flume >> mail # user >> Max Event Size


Copy link to this message
-
RE: Max Event Size
Hi Alex,

I don't see any errors/warning with DEBUG.

Here is the problem: if I tried to read a line larger than 2.5 MB and send via memory channel, the sink lost it and won't receive anything afterwards.

Here is the configuration I tried:

Source: exec. I tried both cat/tail command, and a Java program to read lines from file and print to stdout.
Channel: memory channel and file channel. I think capacity of memory channel is about the size of the queue of events, which won't affect the max size of an event.
Sink: both HDFS and Fill-Roll. They are able to receive short events before large events come.

Can you suggest what to configure? I'm using Flume 1.2.

Thanks,

Kevin

On Nov 23, 2012, at 6:34 PM, Alexander Alten-Lorenz <[EMAIL PROTECTED]> wrote:

Kevin,

This depends on your memory configuration. When you start flume with DEBUG did you got some errors / warning?

Cheers,
 Alex

P.S. http://flume.apache.org/FlumeUserGuide.html#exec-source
On Nov 23, 2012, at 10:29 AM, Lichen <[EMAIL PROTECTED]> wrote:

> Hi all,
>
> I tried to use exec source to tail a file of which each line is few MB. The File-Roll/HDFS sinks received nothing in this case, while work fine with other short lines. Is it because of the maximum size of a single event? What parameter to set if it's configurable.
>
> Thanks
>
> Kevin

--
Alexander Alten-Lorenz
http://mapredit.blogspot.com
German Hadoop LinkedIn Group: http://goo.gl/N8pCF
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB