Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Flume >> mail # user >> Using Flume *only* for tailing?


+
Otis Gospodnetic 2013-06-25, 16:11
+
Bhaskar V. Karambelkar 2013-06-28, 13:26
+
Mahadevappa, Shobha 2013-07-01, 04:56
+
Alexander Alten-Lorenz 2013-06-26, 08:19
+
Otis Gospodnetic 2013-06-27, 23:02
+
Ashish 2013-06-28, 06:11
Copy link to this message
-
Re: Using Flume *only* for tailing?
I don't think that ExecSource was written for that purpose but you *could*
do it. Of course the interface might change at any random time since it's
not a normal use case.

https://github.com/apache/flume/blob/master/flume-ng-core/src/main/java/org/apache/flume/source/ExecSource.java
You'd likely just need to create a channel processor
https://github.com/apache/flume/blob/master/flume-ng-core/src/main/java/org/apache/flume/channel/ChannelProcessor.java

The embedded agent is much different, but might be worth a look as well:

https://github.com/apache/flume/blob/trunk/flume-ng-doc/sphinx/FlumeDeveloperGuide.rst#embedded-agent

Brock
On Thu, Jun 27, 2013 at 6:02 PM, Otis Gospodnetic <
[EMAIL PROTECTED]> wrote:

> Hi Alex,
>
> Sorry, no, I didn't mean tailing of N files into one file.
>
> What I'm wondering is whether I could "steal" a few Flume classes that
> handle the tailing functionality without having to use all of Flume.
> Imagine I want to use Flume's tailing functionality in my app, so I want
> to use Flume as a library.... not being familiar with Flume's APIs, imagine
> if in my app I wanted to do this:
>
> FlumeTail ft = new FlumeTail(new MyLineHandler());
> ft.addFile("/tmp/foo.log");
> ft.exec()
>
> class MyLineHandler extends SomeBaseFlumeLineHandler {
>   void handleLine(String line) {
>     // do whatever I want with that 'line'
>   }
> }
>
> So my questions are:
> * Is something like the above even possible?
> * Could I borrow a few Flume classes just for doing something like the
> above?
>
> Thanks,
> Otis
> ----
> Monitoring for Solr / ElasticSearch / HBase / Hadoop -
> http://sematext.com/spm
>
>   ------------------------------
>  *From:* Alexander Alten-Lorenz <[EMAIL PROTECTED]>
> *To:* [EMAIL PROTECTED]
> *Sent:* Wednesday, June 26, 2013 4:19 AM
> *Subject:* Re: Using Flume *only* for tailing?
>
> Hi Otis,
>
> Do you mean tailing into one file? Maybe 100 exec sources => Flume => one
> text file in a local filesystem?
>
> - Alex
>
> On Jun 25, 2013, at 6:11 PM, Otis Gospodnetic <[EMAIL PROTECTED]>
> wrote:
>
> > Hi,
> >
> > I'm wondering if Flume (its ExecSource?) can be used separately from
> > the rest of Flume?
> >
> > Specifically, I need to tail a log file from a Java app and then do
> > something with each new line.  I just want my Java app to get each new
> > line from tailing and process it in some custom way.  In other words,
> > I don't really need Flume to be involved in anything beyond tailing
> > files.
> >
> > Is that doable?
> >
> > Thanks,
> > Otis
> > --
> > Solr & ElasticSearch Support -- http://sematext.com/
> > Performance Monitoring -- http://sematext.com/spm
>
> --
> Alexander Alten-Lorenz
> http://mapredit.blogspot.com
> German Hadoop LinkedIn Group: http://goo.gl/N8pCF
>
>
>
--
Apache MRUnit - Unit testing MapReduce - http://mrunit.apache.org
+
Israel Ekpo 2013-07-03, 21:03
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB