Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Flume >> mail # user >> Ordering of messages in flume-ng


Copy link to this message
-
Ordering of messages in flume-ng
I've seen a fair number of resources on the web that describe the loose
ordering guarantees that flume offers for messages in the face of
degradation or failures.  But I can't tell what applies to flume-og, and
flume-ng.  Hopefully somebody can help clear up the situation.

In the case of a single agent topology, (source -> FileSystem Channel ->
sink), can messages become out of order?  What situations cause that?

In a multi agent topology, does that answer change?

(Agent 1   Source -> FilesystemChannel -> Avro To Collector)
(Agent 2   Source -> FilesystemChannel -> Avro To Collector)
(Collector Avro from agents -> FilesystemChannel -> final Sink)

And perhaps in an even more complicated setup, with multiple collectors,
does that answer change further?