Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Flume, mail # user - Using Flume *only* for tailing?


+
Otis Gospodnetic 2013-06-25, 16:11
+
Bhaskar V. Karambelkar 2013-06-28, 13:26
+
Mahadevappa, Shobha 2013-07-01, 04:56
+
Alexander Alten-Lorenz 2013-06-26, 08:19
+
Otis Gospodnetic 2013-06-27, 23:02
+
Ashish 2013-06-28, 06:11
Copy link to this message
-
Re: Using Flume *only* for tailing?
Brock Noland 2013-06-27, 23:47
I don't think that ExecSource was written for that purpose but you *could*
do it. Of course the interface might change at any random time since it's
not a normal use case.

https://github.com/apache/flume/blob/master/flume-ng-core/src/main/java/org/apache/flume/source/ExecSource.java
You'd likely just need to create a channel processor
https://github.com/apache/flume/blob/master/flume-ng-core/src/main/java/org/apache/flume/channel/ChannelProcessor.java

The embedded agent is much different, but might be worth a look as well:

https://github.com/apache/flume/blob/trunk/flume-ng-doc/sphinx/FlumeDeveloperGuide.rst#embedded-agent

Brock
On Thu, Jun 27, 2013 at 6:02 PM, Otis Gospodnetic <
[EMAIL PROTECTED]> wrote:

> Hi Alex,
>
> Sorry, no, I didn't mean tailing of N files into one file.
>
> What I'm wondering is whether I could "steal" a few Flume classes that
> handle the tailing functionality without having to use all of Flume.
> Imagine I want to use Flume's tailing functionality in my app, so I want
> to use Flume as a library.... not being familiar with Flume's APIs, imagine
> if in my app I wanted to do this:
>
> FlumeTail ft = new FlumeTail(new MyLineHandler());
> ft.addFile("/tmp/foo.log");
> ft.exec()
>
> class MyLineHandler extends SomeBaseFlumeLineHandler {
>   void handleLine(String line) {
>     // do whatever I want with that 'line'
>   }
> }
>
> So my questions are:
> * Is something like the above even possible?
> * Could I borrow a few Flume classes just for doing something like the
> above?
>
> Thanks,
> Otis
> ----
> Monitoring for Solr / ElasticSearch / HBase / Hadoop -
> http://sematext.com/spm
>
>   ------------------------------
>  *From:* Alexander Alten-Lorenz <[EMAIL PROTECTED]>
> *To:* [EMAIL PROTECTED]
> *Sent:* Wednesday, June 26, 2013 4:19 AM
> *Subject:* Re: Using Flume *only* for tailing?
>
> Hi Otis,
>
> Do you mean tailing into one file? Maybe 100 exec sources => Flume => one
> text file in a local filesystem?
>
> - Alex
>
> On Jun 25, 2013, at 6:11 PM, Otis Gospodnetic <[EMAIL PROTECTED]>
> wrote:
>
> > Hi,
> >
> > I'm wondering if Flume (its ExecSource?) can be used separately from
> > the rest of Flume?
> >
> > Specifically, I need to tail a log file from a Java app and then do
> > something with each new line.  I just want my Java app to get each new
> > line from tailing and process it in some custom way.  In other words,
> > I don't really need Flume to be involved in anything beyond tailing
> > files.
> >
> > Is that doable?
> >
> > Thanks,
> > Otis
> > --
> > Solr & ElasticSearch Support -- http://sematext.com/
> > Performance Monitoring -- http://sematext.com/spm
>
> --
> Alexander Alten-Lorenz
> http://mapredit.blogspot.com
> German Hadoop LinkedIn Group: http://goo.gl/N8pCF
>
>
>
--
Apache MRUnit - Unit testing MapReduce - http://mrunit.apache.org
+
Israel Ekpo 2013-07-03, 21:03