Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Flume >> mail # user >> Can we treat a whole file as a Flume event?


Copy link to this message
-
Re: Can we treat a whole file as a Flume event?
why don't you use directory spooling ?
On Tue, Jan 22, 2013 at 7:15 AM, Henry Ma <[EMAIL PROTECTED]> wrote:

> Hi,
>
> When using Flume to collect log files, we want to just COPY the original
> files from several servers to a central storage (unix file system), not to
> roll up to a big file. Because we must record some messages of the original
> file such as name, host, path, timestamp, etc. Besides, we want to
> guarantee total reliability: no file miss, no file reduplicated.
>
> It seems that, in Source, we must put a whole file (size may be between
> 100KB and 100MB) into a Flume event; and in Sink, we must write each event
> to a single file.
>
> Is it practicable? Thanks!
>
> --
> Best Regards,
> Henry Ma
>

--
Nitin Pawar