Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Flume >> mail # dev >> Flume-ng (v1.3.0) multi tail or tailing directory


Copy link to this message
-
Re: Flume-ng (v1.3.0) multi tail or tailing directory
Hi,

I understand that tail can be applied to multiple files at once but that
will combine all output together and my intension is to keep log data from
different files separate from each other.

Another way is to treat each file to monitor as distinct sources and then
write output to distinct channels (file / memory) and shipping them to
different sinks.

agent.sources = s1 s2 ... sn
agent.channels = fc1 fc2 ... fcn
agent.sinks = sk1 sk2 ... sn

agent.sources.s1.exec  = tail -F -n 0 /var/log/myapp1.log
agent.sources.s2.exec  = tail -F -n 0 /var/log/myapp2.log
.
.
agent.sources.sn.exec  = tail -F -n 0 /var/log/myappn.log

Now s1 ---> fc1 ---> sk1
s2 ---> fc2 ---> sk2
s3 ---> fc3 ---> sk3
..
..
etc.

I am looking for a generic way of achieving the same as explained above if
possible.

Thanks,
- dib
On Thu, Mar 21, 2013 at 2:51 PM, Dibyajyoti Ghosh <[EMAIL PROTECTED]
> wrote:

> Hi flume-dev,
>
> In flume v1.3.0 bundled with cdh-4.2.0 I am aware of exec source where one
> can do 'tail -f' for a single mutable file.
>
> Spool-dir is available but files have to be uniquely named and immutable.
>
> Is there some way I can tail a directory much like multi tail / tailDir
> option in the current flume version?
>
> thanks a lot,
> - dib
>

On Thu, Mar 21, 2013 at 2:51 PM, Dibyajyoti Ghosh <[EMAIL PROTECTED]
> wrote:

> Hi flume-dev,
>
> In flume v1.3.0 bundled with cdh-4.2.0 I am aware of exec source where one
> can do 'tail -f' for a single mutable file.
>
> Spool-dir is available but files have to be uniquely named and immutable.
>
> Is there some way I can tail a directory much like multi tail / tailDir
> option in the current flume version?
>
> thanks a lot,
> - dib
>
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB