Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Flume, mail # user - Can we treat a whole file as a Flume event?


Copy link to this message
-
Can we treat a whole file as a Flume event?
Henry Ma 2013-01-22, 01:45
Hi,

When using Flume to collect log files, we want to just COPY the original
files from several servers to a central storage (unix file system), not to
roll up to a big file. Because we must record some messages of the original
file such as name, host, path, timestamp, etc. Besides, we want to
guarantee total reliability: no file miss, no file reduplicated.

It seems that, in Source, we must put a whole file (size may be between
100KB and 100MB) into a Flume event; and in Sink, we must write each event
to a single file.

Is it practicable? Thanks!

--
Best Regards,
Henry Ma
+
Nitin Pawar 2013-01-22, 05:22
+
Henry Ma 2013-01-22, 06:49
+
Nitin Pawar 2013-01-22, 07:37
+
Mike Percy 2013-01-22, 08:17
+
Roshan Naik 2013-01-22, 23:38
+
Mike Percy 2013-01-23, 02:39
+
Roshan Naik 2013-01-23, 05:23
+
Mike Percy 2013-01-23, 19:53
+
Roshan Naik 2013-01-23, 21:04
+
Mike Percy 2013-01-23, 21:18