Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Flume >> mail # user >> Query regarding readMultiLine in Morphlines config


Copy link to this message
-
Re: Query regarding readMultiLine in Morphlines config
A morphline receives a flume event at a time. What and how much is contained in the flume event is up to you, but flume isn’t really designed to send large events such as whole files or parts of files, it’s designed to send small discrete events, like a log line per event, or similar.

There is no existing command that does what you want. Consider writing a custom morphline command that reads your event and spits out whatever you want, per http://kitesdk.org/docs/current/kite-morphlines/morphlinesReferenceGuide.html#Implementing-your-own-Custom-Command

Having said that, the bottleneck is typically in Lucene inside Solr server, and Flume overheads are insignificant in comparison to that.

Wolfgang.

On Jul 16, 2014, at 2:36 AM, Sanjay Ramanathan <[EMAIL PROTECTED]> wrote: