Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Flume >> mail # user >> How to read multiples files getting continuously updated


Copy link to this message
-
How to read multiples files getting continuously updated
Hi,

I am looking for Flume NG source that can be used for reading many files
which are getting continuously updated.
I trued Spool Dir source but it does not work if file to be read gets
modified.

Here is the scenario:
100 files are getting generated at one time and these files
are continuously  updated for fixed interval say 5 mins, after 5 mins new
100 files get generated and being written again for 5 mins.

Which flume source is most suitable and how it should be used effectively
without any data loss.

Any help is greatly appreciated.
Thanks
Abhijeet Shipure
+
Steve Morin 2013-10-10, 05:52
+
Abhijeet Shipure 2013-10-10, 06:09
+
Steve Morin 2013-10-10, 06:11
+
Abhijeet Shipure 2013-10-10, 06:27
+
Steve Morin 2013-10-10, 06:48
+
DSuiter RDX 2013-10-10, 11:46
+
Paul Chavez 2013-10-10, 16:04
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB