Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Flume >> mail # user >> Can we treat a whole file as a Flume event?


Copy link to this message
-
Can we treat a whole file as a Flume event?
Hi,

When using Flume to collect log files, we want to just COPY the original
files from several servers to a central storage (unix file system), not to
roll up to a big file. Because we must record some messages of the original
file such as name, host, path, timestamp, etc. Besides, we want to
guarantee total reliability: no file miss, no file reduplicated.

It seems that, in Source, we must put a whole file (size may be between
100KB and 100MB) into a Flume event; and in Sink, we must write each event
to a single file.

Is it practicable? Thanks!

--
Best Regards,
Henry Ma
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB