Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Flume >> mail # user >> spooling directory source and variable replacement


+
Frank Maritato 2013-06-03, 20:41
Copy link to this message
-
Re: spooling directory source and variable replacement
Frank,
in spooling directory , it will always pick up all the new files dropped
into directory. Be sure that you do not have files which are still being
written into in the same directory.

In normal use cases, they have a staging directory where they have current
on going log writing. And then you can use logrotate to move the files from
your log directory to spooling directory. Spooling directory requires the
directory from which it needs to pick up files and you can not put a file
name in the config (as far as I know)

If you want to concentrate on those filenames only then I would suggest to
only move those files into the spooling directory.
On Tue, Jun 4, 2013 at 2:11 AM, Frank Maritato <[EMAIL PROTECTED]>wrote:

>  Hi All,
>
>  The application I'd like to grab log files from is rotating them into
> subdirectories by time stamp. For example,
>
>  /mnt/remote/application_name/yyyy/mm/dd/hh/[filename]-[timestamp].gz
>
>  Is there any way to configure the spooling directory source in flume
> with time variables such that it can find these files? Or is there a better
> way to do this?
>
>  Thanks
>  --
> Frank Maritato
>
>
>
>
>
--
Nitin Pawar
+
Phil Scala 2013-06-06, 18:20
+
Frank Maritato 2013-06-06, 18:29
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB