Hi All,

I am a newbie to flume. I have started using it for log aggregation in my company.

I have been desperately trying to make the hostname interceptor work for me , but i am not getting the hostName in the final aggregated log file.

The conf file at my AGGREGATOR machine is

# Describe/configure source1
agent1.sources.source1.type = avro
agent1.sources.source1.bind = A.B.C.D
agent1.sources.source1.port = 44444

agent1.sources.source1.interceptors = host_int inter1
agent1.sources.source1.interceptors.host_int.type = host
agent1.sources.source1.interceptors.host_int.hostHeader = hostname

agent1.sources.source1.interceptors.inter1.type = timestamp
# Describe sink1
agent1.sinks.sink1.type = FILE_ROLL
agent1.sinks.sink1.sink.directory=/home/ashutosh/flumeLogs
agent1.sinks.sink1.sink.rollInterval=0
The configuration  file at my  agents are .  

agent1.sources.source1.type = exec
agent1.sources.source1.command = tail -F /mnt1/logs/LOGGER.log
My expectation was that in the Aggregator machine, whatever logs i get, will have the hostname in the beginning. But i am getting the actual string which was generated by agent.

Please help.
Regards
Ashutosh

Hi,

The interceptor is probably working as expected. The reason for this is that the serializer that the FILE_ROLL sink uses by default is the BodyText serializer which simply serializes the body. The hostname is in the header though, so you need to write a serializer that can write the header out in the format you want. See : http://flume.apache.org/FlumeUserGuide.html#file-roll-sink. You can write your own serializer and then supply it through the configuration.  
Thanks
Hari
--
Hari Shreedharan
On Wednesday, August 29, 2012 at 1:57 AM, Ashutosh Panigrahi wrote:

> Hi All,
>
> I am a newbie to flume. I have started using it for log aggregation in my company.
>
> I have been desperately trying to make the hostname interceptor work for me , but i am not getting the hostName in the final aggregated log file.
>
> The conf file at my AGGREGATOR machine is
>
> # Describe/configure source1
> agent1.sources.source1.type = avro
> agent1.sources.source1.bind = A.B.C.D
> agent1.sources.source1.port = 44444
>
> agent1.sources.source1.interceptors = host_int inter1
> agent1.sources.source1.interceptors.host_int.type = host
> agent1.sources.source1.interceptors.host_int.hostHeader = hostname
>
> agent1.sources.source1.interceptors.inter1.type = timestamp
>
>
> # Describe sink1
> agent1.sinks.sink1.type = FILE_ROLL
> agent1.sinks.sink1.sink.directory=/home/ashutosh/flumeLogs
> agent1.sinks.sink1.sink.rollInterval=0
>
>
>
> The configuration file at my agents are .
>
> agent1.sources.source1.type = exec
> agent1.sources.source1.command = tail -F /mnt1/logs/LOGGER.log
>
>
> My expectation was that in the Aggregator machine, whatever logs i get, will have the hostname in the beginning. But i am getting the actual string which was generated by agent.
>
> Please help.
>
>
> Regards
> Ashutosh

Worth trying out the AVRO_EVENT serializer which comes out of the box,
which writes the data in Avro container format (including all headers).

Regards,
Mike

On Wed, Aug 29, 2012 at 2:11 AM, Hari Shreedharan <[EMAIL PROTECTED]
> wrote:

> Hi,
>
> The interceptor is probably working as expected. The reason for this is
> that the serializer that the FILE_ROLL sink uses by default is the BodyText
> serializer which simply serializes the body. The hostname is in the header
> though, so you need to write a serializer that can write the header out in
> the format you want. See :
> http://flume.apache.org/FlumeUserGuide.html#file-roll-sink. You can write
> your own serializer and then supply it through the configuration.
>
>
> Thanks
> Hari
> --
> Hari Shreedharan
>
>
> On Wednesday, August 29, 2012 at 1:57 AM, Ashutosh Panigrahi wrote:
>
> > Hi All,
> >
> > I am a newbie to flume. I have started using it for log aggregation in
> my company.
> >
> > I have been desperately trying to make the hostname interceptor work for
> me , but i am not getting the hostName in the final aggregated log file.
> >
> > The conf file at my AGGREGATOR machine is
> >
> > # Describe/configure source1
> > agent1.sources.source1.type = avro
> > agent1.sources.source1.bind = A.B.C.D
> > agent1.sources.source1.port = 44444
> >
> > agent1.sources.source1.interceptors = host_int inter1
> > agent1.sources.source1.interceptors.host_int.type = host
> > agent1.sources.source1.interceptors.host_int.hostHeader = hostname
> >
> > agent1.sources.source1.interceptors.inter1.type = timestamp
> >
> >
> > # Describe sink1
> > agent1.sinks.sink1.type = FILE_ROLL
> > agent1.sinks.sink1.sink.directory=/home/ashutosh/flumeLogs
> > agent1.sinks.sink1.sink.rollInterval=0
> >
> >
> >
> > The configuration file at my agents are .
> >
> > agent1.sources.source1.type = exec
> > agent1.sources.source1.command = tail -F /mnt1/logs/LOGGER.log
> >
> >
> > My expectation was that in the Aggregator machine, whatever logs i get,
> will have the hostname in the beginning. But i am getting the actual string
> which was generated by agent.
> >
> > Please help.
> >
> >
> > Regards
> > Ashutosh
>
>
>
>