Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Flume, mail # user - Can we treat a whole file as a Flume event?


Copy link to this message
-
Re: Can we treat a whole file as a Flume event?
Henry Ma 2013-01-22, 06:49
As far as I know, Directory Spooling Source will send the file line by line
as an event, and File Roll Sink will receive these lines and roll up to a
big file by a fixed interval. Is it right, and can we config it to send the
whole file as an event?
On Tue, Jan 22, 2013 at 1:22 PM, Nitin Pawar <[EMAIL PROTECTED]>wrote:

> why don't you use directory spooling ?
>
>
> On Tue, Jan 22, 2013 at 7:15 AM, Henry Ma <[EMAIL PROTECTED]> wrote:
>
>> Hi,
>>
>> When using Flume to collect log files, we want to just COPY the original
>> files from several servers to a central storage (unix file system), not to
>> roll up to a big file. Because we must record some messages of the original
>> file such as name, host, path, timestamp, etc. Besides, we want to
>> guarantee total reliability: no file miss, no file reduplicated.
>>
>> It seems that, in Source, we must put a whole file (size may be between
>> 100KB and 100MB) into a Flume event; and in Sink, we must write each event
>> to a single file.
>>
>> Is it practicable? Thanks!
>>
>> --
>> Best Regards,
>> Henry Ma
>>
>
>
>
> --
> Nitin Pawar
>

--
Best Regards,
Henry Ma