Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Flume >> mail # user >> buffermaxlinelength restriction in spool directory


Copy link to this message
-
buffermaxlinelength restriction in spool directory
Hi,
   I am using spool directory source, but am facing problem because of the
buffermaxlinelength.

   My use case is, that I am storing some log files in the spool directory,
however the system which is producing the log files really don't know about
logging standards. There are log records of size MB. My requirement is to
store these files into hadoop, so I thought of using flume. So in short
whenever the spool directory source reads a record greater than the
bufferMaxLineLength it throws an exception and then the process is stuck as
the same message is tried over and over again. I don't want to increase the
bufferMaxLineLegth to a ridiculous large value just because of very few big
messages as it will require to increase the heap size (keeping the channel
capacity and transaction capacity constant).

   Is there a way other than specifying befferMaxLineLength to read these
unusually large messages? or any suggestions to gracefully dropping such
messages while using spool directory source?

/Ehsan
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB