Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Flume >> mail # user >> How to read multiples files getting continuously updated

Copy link to this message
How to read multiples files getting continuously updated

I am looking for Flume NG source that can be used for reading many files
which are getting continuously updated.
I trued Spool Dir source but it does not work if file to be read gets

Here is the scenario:
100 files are getting generated at one time and these files
are continuously  updated for fixed interval say 5 mins, after 5 mins new
100 files get generated and being written again for 5 mins.

Which flume source is most suitable and how it should be used effectively
without any data loss.

Any help is greatly appreciated.
Abhijeet Shipure
Steve Morin 2013-10-10, 05:52
Abhijeet Shipure 2013-10-10, 06:09
Steve Morin 2013-10-10, 06:11
Abhijeet Shipure 2013-10-10, 06:27
Steve Morin 2013-10-10, 06:48
DSuiter RDX 2013-10-10, 11:46
Paul Chavez 2013-10-10, 16:04