Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Flume >> mail # user >> Reliability in Flume


Copy link to this message
-
Re: Reliability in Flume
Hi Henry,

Just to add to Mike's response:

When used with secure channels(mainly file channel) and with transports
that can be rolled back(avro), message delivery is
guarranteed(eventually). The only way you can lose data is for a part of
the chain to be permanently removed: HD failure or removal of the
physical hardware.

Prevention of data duplication has never been an objective of flume,
though it is uncommon in a properly configured setup. The larger your  
batch sizes are, the more duplication you may get with each partial
failure. Similarly ordered arrival of data is not guarranteed. The best
way to address these two issues, if it is a concern, is to run a
map-reduce task or similar to reduce to unique rows and/or reorder.

On 01/24/2013 12:26 PM, Henry Ma wrote:
> Dear Flume developers and users,
>
> I understand that Flume NG uses channel-based transactions to
> guarantee reliable message delivery between agents. But in
> some extreme failure scenes, will Flume keep total Reliability? I have
> thought of these scenes below.
>
> 1. In transactions between agent, what will happen if the receiving
> agent process down just after it commits its put transaction and
> before sends the success indication to the sending agent? Will the
> sending agent send the same event again when the receiving agent
> recovers, and cause data duplication?
>
> 2. In the communication between the client (data source, sending data
> to the first-hop agent) and the first-hop agent, what will happen if
> the agent process down just after it receives the event and before
> saves to its channel? Will it cause data loss?
>
> 3. In the communication between the final-hup agent and the storage
> system (such as MySQL, HDFS, file system, etc.), what happened if the
> agent down before it commits the saving transaction but has saved some
> data in the storage? Will this cause data duplication after the
> recover of the agent?
>
> Thank you very much!
> --
> Best Regards,
> Henry Ma