Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Flume >> mail # user >> syslog source - sinks without datetime/hostname


+
Filip Slunecko 2012-10-16, 21:27
+
Roshan Naik 2012-10-16, 21:32
+
Hari Shreedharan 2012-10-16, 22:00
+
Filip Slunecko 2012-10-16, 22:25
Copy link to this message
-
Re: syslog source - sinks without datetime/hostname
See the code for the serializers here:
https://git-wip-us.apache.org/repos/asf?p=flume.git;a=tree;f=flume-ng-core/src/main/java/org/apache/flume/serialization;h=fcc07339b3cf0f5b8d1a75e978ffc1edbab28bfe;hb=HEAD
You can use one of these as an example to write your own.

The configuration documentation for HDFS Sink is here: http://flume.apache.org/FlumeUserGuide.html#hdfs-sink
You can specify a custom serializer by specifying its builder class in the config param "serializer".
Thanks,
Hari

--
Hari Shreedharan
On Tuesday, October 16, 2012 at 3:25 PM, Filip Slunecko wrote:

> @Roshan: thanks for suggestion, I will look into it.
>
> @Hari: I tried to google it up and there is not much about it. I will
> look into it tomorrow and let you know (It' s too late for serious
> work in my time realm :) )
>
> Thanks both of you for your quick response and help!
>
> Filip
>
> On Wed, Oct 17, 2012 at 12:00 AM, Hari Shreedharan
> <[EMAIL PROTECTED] (mailto:[EMAIL PROTECTED])> wrote:
> > Hi Filip,
> >
> > The reason for this is that the Text serializer will only serialized the
> > body of the event, and the syslog sources write the body of the syslog event
> > into the body of the flume event. The hostname/timestamp/severity etc are
> > added into the Flume Event headers. You could simply write a serializer
> > which writes out this information in the same format as you expect and you
> > will be able to see the headers in the files. You could use the Avro
> > serializer to serialize it into avro too, which will make sure the headers
> > are also written out.
> >
> > Hope this helps.
> >
> >
> > Hari
> >
> > --
> > Hari Shreedharan
> >
> > On Tuesday, October 16, 2012 at 2:27 PM, Filip Slunecko wrote:
> >
> > Hi,
> >
> > I am trying to use syslog source and sink it to hdfs or fileroller.
> > Everything is working, but "saved" logs are without timestamp and
> > hostname information.
> > Is it possible to force flume-ng to dump those information from syslog
> > header togather with body lines?
> >
> > I am using flume-ng-agent-1.2.0+24.4-1.noarch from Cloudera repository.
> >
> > Thanks,
> >
> > Filip

+
Roshan Naik 2012-10-16, 22:48
+
Hari Shreedharan 2012-10-16, 23:01
+
Bhaskar V. Karambelkar 2012-10-16, 23:24
+
Filip Slunecko 2012-11-03, 13:54