Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Flume, mail # user - Syslog-ng, Flume (old gen) and metadata


Copy link to this message
-
Re: Syslog-ng, Flume (old gen) and metadata
Brock Noland 2012-10-30, 15:12
Hi,

I don't actually know the answer to your question, hopefully someone
else will help with that, but I have some hopefully useful comments
below.

On Mon, Oct 29, 2012 at 10:51 AM, Bertrand Dechoux <[EMAIL PROTECTED]> wrote:
> Hi,
>
> I would like to deepen my understanding of syslog-ng and flume integration.
>
> I had initial three questions :
> 1) Does flume timestamp is the same as the received syslog-ng event
> timestamp?
> 2) Are the syslog-ng metadata kept by flume as extra metadata (> syslog-ng
> OSE 3.0)?
> 3) Are the extra metadata (and potientially syslog-ng ones) usable with the
> collector sink template?
>
> From what I understand :
> 1) Yes, acoording the cookbook. "use the timestamp found in the original
> data,"
> http://archive.cloudera.com/cdh/3/flume/Cookbook/

That is Flume OG which is still available but as far as I am aware no
one is actively taking that forward. Flume NG is actively being worked
on.  The Cloudera link for NG is here:
http://archive.cloudera.com/cdh/3/flume-ng-1.2.0-cdh3u5/ and the
apache webpage for flume is all NG content as far as I am aware:

http://flume.apache.org/

> 2) No but I haven't found a clear answer about it.
> 3) No but I haven't found a clear answer about it.
>
> Could someone confirm me that?
> I would believe 2) and 3) be generally asked features for those integrating
> existent syslog-ng system with Flume but maybe it isn't. Or maybe there are
> third party libraries for that? What is the status of Flume-ng with regards
> to the same questions?
>
> Regards
>
> Bertrand

--
Apache MRUnit - Unit testing MapReduce - http://incubator.apache.org/mrunit/