Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Flume >> mail # user >> Using Flume *only* for tailing?


Copy link to this message
-
Re: Using Flume *only* for tailing?
Otis,

I have not done this, but looking at test case would show the trick.

https://github.com/apache/flume/blob/master/flume-ng-core/src/test/java/org/apache/flume/source/TestExecSource.java

>From the very little knowledge of Flume I have, you could work out a util
class like the Test class, and implement a custom channel with the line
handler and do what is needed. I would recommend using interceptors for
line/Event handling

Might be some Transactional issues, but as of now not able provide further
insight into this.

HTH !

On Fri, Jun 28, 2013 at 4:32 AM, Otis Gospodnetic <
[EMAIL PROTECTED]> wrote:

> Hi Alex,
>
> Sorry, no, I didn't mean tailing of N files into one file.
>
> What I'm wondering is whether I could "steal" a few Flume classes that
> handle the tailing functionality without having to use all of Flume.
> Imagine I want to use Flume's tailing functionality in my app, so I want
> to use Flume as a library.... not being familiar with Flume's APIs, imagine
> if in my app I wanted to do this:
>
> FlumeTail ft = new FlumeTail(new MyLineHandler());
> ft.addFile("/tmp/foo.log");
> ft.exec()
>
> class MyLineHandler extends SomeBaseFlumeLineHandler {
>   void handleLine(String line) {
>     // do whatever I want with that 'line'
>   }
> }
>
> So my questions are:
> * Is something like the above even possible?
> * Could I borrow a few Flume classes just for doing something like the
> above?
>
> Thanks,
> Otis
> ----
> Monitoring for Solr / ElasticSearch / HBase / Hadoop -
> http://sematext.com/spm
>
>   ------------------------------
>  *From:* Alexander Alten-Lorenz <[EMAIL PROTECTED]>
> *To:* [EMAIL PROTECTED]
> *Sent:* Wednesday, June 26, 2013 4:19 AM
> *Subject:* Re: Using Flume *only* for tailing?
>
> Hi Otis,
>
> Do you mean tailing into one file? Maybe 100 exec sources => Flume => one
> text file in a local filesystem?
>
> - Alex
>
> On Jun 25, 2013, at 6:11 PM, Otis Gospodnetic <[EMAIL PROTECTED]>
> wrote:
>
> > Hi,
> >
> > I'm wondering if Flume (its ExecSource?) can be used separately from
> > the rest of Flume?
> >
> > Specifically, I need to tail a log file from a Java app and then do
> > something with each new line.  I just want my Java app to get each new
> > line from tailing and process it in some custom way.  In other words,
> > I don't really need Flume to be involved in anything beyond tailing
> > files.
> >
> > Is that doable?
> >
> > Thanks,
> > Otis
> > --
> > Solr & ElasticSearch Support -- http://sematext.com/
> > Performance Monitoring -- http://sematext.com/spm
>
> --
> Alexander Alten-Lorenz
> http://mapredit.blogspot.com
> German Hadoop LinkedIn Group: http://goo.gl/N8pCF
>
>
>
--
thanks
ashish

Blog: http://www.ashishpaliwal.com/blog
My Photo Galleries: http://www.pbase.com/ashishpaliwal
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB