Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Flume >> mail # user >> How to read multiples files getting continuously updated


Copy link to this message
-
How to read multiples files getting continuously updated
Hi,

I am looking for Flume NG source that can be used for reading many files
which are getting continuously updated.
I trued Spool Dir source but it does not work if file to be read gets
modified.

Here is the scenario:
100 files are getting generated at one time and these files
are continuously  updated for fixed interval say 5 mins, after 5 mins new
100 files get generated and being written again for 5 mins.

Which flume source is most suitable and how it should be used effectively
without any data loss.

Any help is greatly appreciated.
Thanks
Abhijeet Shipure
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB