Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Flume >> mail # user >> Can we treat a whole file as a Flume event?


Copy link to this message
-
Re: Can we treat a whole file as a Flume event?
As far as I know, Directory Spooling Source will send the file line by line
as an event, and File Roll Sink will receive these lines and roll up to a
big file by a fixed interval. Is it right, and can we config it to send the
whole file as an event?
On Tue, Jan 22, 2013 at 1:22 PM, Nitin Pawar <[EMAIL PROTECTED]>wrote:

> why don't you use directory spooling ?
>
>
> On Tue, Jan 22, 2013 at 7:15 AM, Henry Ma <[EMAIL PROTECTED]> wrote:
>
>> Hi,
>>
>> When using Flume to collect log files, we want to just COPY the original
>> files from several servers to a central storage (unix file system), not to
>> roll up to a big file. Because we must record some messages of the original
>> file such as name, host, path, timestamp, etc. Besides, we want to
>> guarantee total reliability: no file miss, no file reduplicated.
>>
>> It seems that, in Source, we must put a whole file (size may be between
>> 100KB and 100MB) into a Flume event; and in Sink, we must write each event
>> to a single file.
>>
>> Is it practicable? Thanks!
>>
>> --
>> Best Regards,
>> Henry Ma
>>
>
>
>
> --
> Nitin Pawar
>

--
Best Regards,
Henry Ma
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB