Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Flume >> mail # user >> Syslog-ng, Flume (old gen) and metadata


Copy link to this message
-
Re: Syslog-ng, Flume (old gen) and metadata
Hi,

I don't actually know the answer to your question, hopefully someone
else will help with that, but I have some hopefully useful comments
below.

On Mon, Oct 29, 2012 at 10:51 AM, Bertrand Dechoux <[EMAIL PROTECTED]> wrote:
> Hi,
>
> I would like to deepen my understanding of syslog-ng and flume integration.
>
> I had initial three questions :
> 1) Does flume timestamp is the same as the received syslog-ng event
> timestamp?
> 2) Are the syslog-ng metadata kept by flume as extra metadata (> syslog-ng
> OSE 3.0)?
> 3) Are the extra metadata (and potientially syslog-ng ones) usable with the
> collector sink template?
>
> From what I understand :
> 1) Yes, acoording the cookbook. "use the timestamp found in the original
> data,"
> http://archive.cloudera.com/cdh/3/flume/Cookbook/

That is Flume OG which is still available but as far as I am aware no
one is actively taking that forward. Flume NG is actively being worked
on.  The Cloudera link for NG is here:
http://archive.cloudera.com/cdh/3/flume-ng-1.2.0-cdh3u5/ and the
apache webpage for flume is all NG content as far as I am aware:

http://flume.apache.org/

> 2) No but I haven't found a clear answer about it.
> 3) No but I haven't found a clear answer about it.
>
> Could someone confirm me that?
> I would believe 2) and 3) be generally asked features for those integrating
> existent syslog-ng system with Flume but maybe it isn't. Or maybe there are
> third party libraries for that? What is the status of Flume-ng with regards
> to the same questions?
>
> Regards
>
> Bertrand

--
Apache MRUnit - Unit testing MapReduce - http://incubator.apache.org/mrunit/
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB