Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Flume >> mail # user >> doubt in exec source specifically in tail -F


Copy link to this message
-
doubt in exec source specifically in tail -F
Hi

In Flume-ng is there any way using exec (tail -F) as the source to get
only the new lines  which are being added to the log file ?
(i.e. there is a growing log file and we want to transfer all the logs
using flume
without duplication of logs)

I understand if something fails and as tail doesn't maintain state we
will have duplicates.
But we are not considering failovers as of now.

So I think "tail -F" is useful only in scenarios where sink or any
intermediate
agent can remove duplicates. Is it correct?

But as tail looks like quite a popular source in flume I thought I might
be missing
something.....
Presently using "tail -F <file>" as the source to read from the log file
leads to
scenarios like this:

1. If file has not  changed for a while, but tail still tails file every
second and then prints the same lines again (depending upon -n option)
2. Even if file grows then using tail we can't quite control which lines
we want?

Regards,
Jagadish
+
Brock Noland 2012-07-28, 14:18
+
Gumnaam Sur 2012-07-28, 15:17
+
khadar basha 2012-07-28, 16:56
+
Brock Noland 2012-07-28, 18:54
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB