Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Flume >> mail # user >> Flume as event/alerts collector


Copy link to this message
-
Re: Flume as event/alerts collector
Hi Karim

On 10/13/2012 08:03 AM, Zamani, Karim wrote:
>
> Hi,
>
> Is Flume (or Flume-NG) suitable for collecting events such as alters
> generated by various nodes in a distributed architecture to a central
> location?
>
> I read in the wiki FAQ that Flume was not designed to be a message
> passing or eventing framework due to issues like message duplication,
> ordering, etc.
>
Flume NG only guarrantees delivery(so long as you use durable channels
on every hop). Avoiding duplication is not a hard objective, and it is
possible. Same applies for message ordering.

That said, not all component will result in message order getting mixed
or duplicates. If you deliver messages in order to flume and sink them
without batching, along a single path, they should be in order and
without duplicates. However this puts some severe restrictions on
multiple delivery paths for redundancy and batching(not batching the
sink at all is going to hurt performance), so I couldn't claim flume is
ideal for a system where ordering/avoiding duplication is  necessary.

> Does Flume-NG have any enhancements to support message passing better?
>
> Thanks!
>
> Karim
>

NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB