Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Flume, mail # user - How to read multiples files getting continuously updated


Copy link to this message
-
How to read multiples files getting continuously updated
Abhijeet Shipure 2013-10-10, 05:33
Hi,

I am looking for Flume NG source that can be used for reading many files
which are getting continuously updated.
I trued Spool Dir source but it does not work if file to be read gets
modified.

Here is the scenario:
100 files are getting generated at one time and these files
are continuously  updated for fixed interval say 5 mins, after 5 mins new
100 files get generated and being written again for 5 mins.

Which flume source is most suitable and how it should be used effectively
without any data loss.

Any help is greatly appreciated.
Thanks
Abhijeet Shipure
+
Steve Morin 2013-10-10, 05:52
+
Abhijeet Shipure 2013-10-10, 06:09
+
Steve Morin 2013-10-10, 06:11
+
Abhijeet Shipure 2013-10-10, 06:27
+
Steve Morin 2013-10-10, 06:48
+
DSuiter RDX 2013-10-10, 11:46
+
Paul Chavez 2013-10-10, 16:04