Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Flume, mail # user - Does File Channel write first?

Copy link to this message
Re: Does File Channel write first?
Otis Gospodnetic 2014-02-26, 22:03

Thanks for the link, Hari!

It looks like the only way to avoid having Flume write data to be sent to
Sink on disk first is by using
https://issues.apache.org/jira/browse/FLUME-1227 , once it's committed.

I have a few related questions:

* How/when does Flume delete data from FileChannel?
* Does it delete individual "records" as soon as a "record" is sent out?
* Does it periodically purge batches of data?
* Is there a notion of TTL, like in Kafka, where data is not removed
explicitly by its consumer, but is deleted by Kafka Broker after some TTL?

* What happens with data that could not be sent?
* I know there is a retry and backoff mechanism.  But does Flume at some
point give up on trying to send some (old) piece of data out because it's
tried > N times or for > M seconds?

Performance Monitoring * Log Analytics * Search Analytics
Solr & Elasticsearch Support * http://sematext.com/
On Wed, Feb 26, 2014 at 2:15 PM, Hari Shreedharan <[EMAIL PROTECTED]