Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Flume >> mail # user >> Would someone please comment on Tail Source in NG?


+
Chris Neal 2012-08-29, 16:18
+
Juhani Connolly 2012-08-30, 01:51
+
Patrick Wendell 2012-08-30, 05:07
+
Chris Neal 2012-08-30, 14:45
+
Steve Johnson 2012-08-30, 15:06
+
Patrick Wendell 2012-09-02, 21:45
+
Steve Johnson 2012-09-05, 14:26
+
Patrick Wendell 2012-09-05, 16:47
Copy link to this message
-
Re: Would someone please comment on Tail Source in NG?
Juhani,

Thanks for the response :)  I'll take a look at the jira.
I like your solution.  I might end up going a route similar.

Chris

On Wed, Aug 29, 2012 at 8:51 PM, Juhani Connolly <
[EMAIL PROTECTED]> wrote:

>  Hi Chris,
>
> A few months back I actually ported the original flumes tail source, but
> it was decided(and I agree with the reasoning) not to include it for a
> number of reasons, which can be seen on the original ticket at
> https://issues.apache.org/jira/browse/FLUME-931 . One of the big ones is
> the fact that java cannot access inode information.
>
> What we do is have a python program that tracks the files in a directory
> and then sends the data using the scribe format to the ScribeSource(we were
> using scribe until switching to flume, so are just using our ingest system
> from then). This allows for the freedom to customize the ingest to our own
> expectations, and we write checkpoints of how far we have tailed. You could
> write this in whatever language you're comfortable with and pass the data
> via avro or thrift.
>
>
> On 08/30/2012 01:18 AM, Chris Neal wrote:
>
> Hey guys,
>
>  I'm sure this is not a new question, but I haven't found an answer in my
> searches.  I'm curious why there is as of yet no Tail Source with NG?  It
> seems one of the most common use cases for Flume is to tail a log file and
> dump it "somewhere".  Given that, it sure would seem that a Tail Source
> would be one of the first sources that gets written with a new version.
>
> I know about all the other ways to implement something *like* a Tail
> Source:  Exec Source, AVRO, Log4Jappender...  and unfortunately they all
> have limitations with regards to either functionality or
> reliability/recoverability.
>
>  What am I missing here?
>
>  Is there any work being done on a Tail Source for NG?
>
>  I promise I'm not complaining, just trying to understand the logic. :)
>
>  Much appreciated.
> Chris
>
>
>
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB