Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Flume >> mail # user >> Ordering of messages in flume-ng


Copy link to this message
-
Ordering of messages in flume-ng
I've seen a fair number of resources on the web that describe the loose
ordering guarantees that flume offers for messages in the face of
degradation or failures.  But I can't tell what applies to flume-og, and
flume-ng.  Hopefully somebody can help clear up the situation.

In the case of a single agent topology, (source -> FileSystem Channel ->
sink), can messages become out of order?  What situations cause that?

In a multi agent topology, does that answer change?

(Agent 1   Source -> FilesystemChannel -> Avro To Collector)
(Agent 2   Source -> FilesystemChannel -> Avro To Collector)
(Collector Avro from agents -> FilesystemChannel -> final Sink)

And perhaps in an even more complicated setup, with multiple collectors,
does that answer change further?

 
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB