Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Flume >> mail # user >> syslog source - sinks without datetime/hostname


+
Filip Slunecko 2012-10-16, 21:27
+
Roshan Naik 2012-10-16, 21:32
+
Hari Shreedharan 2012-10-16, 22:00
+
Filip Slunecko 2012-10-16, 22:25
+
Hari Shreedharan 2012-10-16, 22:36
Copy link to this message
-
Re: syslog source - sinks without datetime/hostname
Hari,
   wouldn't  a custom interceptor be more logical ?
-roshan
On Tue, Oct 16, 2012 at 3:36 PM, Hari Shreedharan <[EMAIL PROTECTED]
> wrote:

>  See the code for the serializers here:
>
> https://git-wip-us.apache.org/repos/asf?p=flume.git;a=tree;f=flume-ng-core/src/main/java/org/apache/flume/serialization;h=fcc07339b3cf0f5b8d1a75e978ffc1edbab28bfe;hb=HEAD
> You can use one of these as an example to write your own.
>
> The configuration documentation for HDFS Sink is here:
> http://flume.apache.org/FlumeUserGuide.html#hdfs-sink
> You can specify a custom serializer by specifying its builder class in the
> config param "serializer".
>
>
> Thanks,
> Hari
>
> --
> Hari Shreedharan
>
> On Tuesday, October 16, 2012 at 3:25 PM, Filip Slunecko wrote:
>
> @Roshan: thanks for suggestion, I will look into it.
>
> @Hari: I tried to google it up and there is not much about it. I will
> look into it tomorrow and let you know (It' s too late for serious
> work in my time realm :) )
>
> Thanks both of you for your quick response and help!
>
> Filip
>
> On Wed, Oct 17, 2012 at 12:00 AM, Hari Shreedharan
> <[EMAIL PROTECTED]> wrote:
>
> Hi Filip,
>
> The reason for this is that the Text serializer will only serialized the
> body of the event, and the syslog sources write the body of the syslog
> event
> into the body of the flume event. The hostname/timestamp/severity etc are
> added into the Flume Event headers. You could simply write a serializer
> which writes out this information in the same format as you expect and you
> will be able to see the headers in the files. You could use the Avro
> serializer to serialize it into avro too, which will make sure the headers
> are also written out.
>
> Hope this helps.
>
>
> Hari
>
> --
> Hari Shreedharan
>
> On Tuesday, October 16, 2012 at 2:27 PM, Filip Slunecko wrote:
>
> Hi,
>
> I am trying to use syslog source and sink it to hdfs or fileroller.
> Everything is working, but "saved" logs are without timestamp and
> hostname information.
> Is it possible to force flume-ng to dump those information from syslog
> header togather with body lines?
>
> I am using flume-ng-agent-1.2.0+24.4-1.noarch from Cloudera repository.
>
> Thanks,
>
> Filip
>
>
>
+
Hari Shreedharan 2012-10-16, 23:01
+
Bhaskar V. Karambelkar 2012-10-16, 23:24
+
Filip Slunecko 2012-11-03, 13:54
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB