+ user@

Thamatam,

The Log4J appender adds the date, log level and logger name to the flume event headers and the text of the log event to the flume event body. The reason the log level and time are missing is that these are in the headers and the text serializer does not serialize the headers.

To write to a file or HDFS, please use a Serializer together with the RollingFileSink or HDFSEventSink. Please take a look at the plain text serializer or Avro serializer to understand this better.

Thanks,
Hari

--
Hari Shreedharan
On Saturday, July 28, 2012 at 5:47 PM, thamatam Jayaprakash wrote:

> Hi Hari,
>
>
> Actually im unable to send to this mail to the user and dev group so, im mailing to you .
>
> Could you pls point me where im going wrong.
> Please suggest me which log appender need to use for custom logging pattern and appender.
>
> Im working on Flume 1.1.0 and 1.2.0 . We are not able set log pattern and
> We are using log4jappender
> log4j.appender.flume=org.apache.flume.clients.log4jappender.Log4jAppender
>
> but we are getting plain test
>
> Example if i log following mssages :
>
> 17:42:55,928  INFO SimpleJdbcServlet:69 - doGet of SimpleJdbcServlet ended...
> 17:43:03,489  INFO HelloServlet:29 - HelloServlet of doGet started...
> 17:43:03,489  INFO HelloServlet:33 -
>  Hello from Simple Servlet
> 17:43:03,489  INFO HelloServlet:35 - HelloServlet of doGet end...
> 17:47:46,000  INFO HelloServlet:29 - HelloServlet of doGet started...
> 17:47:46,001  INFO HelloServlet:33 -
>  Hello from Simple Servlet
> 17:47:46,001  INFO HelloServlet:35 - HelloServlet of doGet end...
>
> Using Flume in Hadoop im getting only the following logs:
>
> doGet of SimpleJdbcServlet ended...
> HelloServlet of doGet started...
>
> HelloServlet of doGet end...
> HelloServlet of doGet started...
>
>
>
> Thanks in advance.
>
> --
> JP
>
>
>
> --
> Jayaprakash

Hi  Hari,

i am also facing same problem.  Here is my configuration.
flume-conf.properties file
=================agent2Test1.sources = seqGenSrc
agent2Test1.channels = memoryChannel
agent2Test1.sinks = loggerSink

# For each one of the sources, the type is defined
*agent2Test1.sources.seqGenSrc.type = avro*
agent2Test1.sources.seqGenSrc.bind=localhost
agent2Test1.sources.seqGenSrc.port=41414

# interceptors for host and date
agent2Test1.sources.seqGenSrc.interceptors = time hostInterceptor
agent2Test1.sources.seqGenSrc.interceptors.hostInterceptor.type org.apache.flume.interceptor.HostInterceptor$Builder
agent2Test1.sources.seqGenSrc.interceptors.hostInterceptor.hostHeader = host
agent2Test1.sources.seqGenSrc.interceptors.hostInterceptor.useIP = false
agent2Test1.sources.seqGenSrc.interceptors.hostInterceptor.hostHeader.preserveExisting
= false
agent2Test1.sources.seqGenSrc.interceptors.time.type org.apache.flume.interceptor.TimestampInterceptor$Builder

# The channel can be defined as follows.
agent2Test1.sources.seqGenSrc.channels = memoryChannel

# Each sink's type must be defined
agent2Test1.sinks.loggerSink.type = hdfs
agent2Test1.sinks.loggerSink.hdfs.path hdfs://hadoopHost:8020/data/%Y/%m/%d/%{host}/Logs

agent2Test1.sinks.loggerSink.hdfs.fileType = DataStream

#Specify the channel the sink should use
agent2Test1.sinks.loggerSink.channel = memoryChannel

# Each channel's type is defined.
agent2Test1.channels.memoryChannel.type = memory

# Other config values specific to each type of channel(sink or source)
# can be defined as well
# In this case, it specifies the capacity of the memory channel
agent2Test1.channels.memoryChannel.capacity = 1000

Sample java program to generate the log message:
====================================

package com.test;
import org.apache.flume.clients.log4jappender.Log4jAppender;
import org.apache.log4j.Logger;
import org.apache.log4j.MDC;
import org.apache.log4j.PatternLayout;

import java.util.UUID;
public class Main {
    static Logger log = Logger.getLogger(Main.class);

    public static void main(String[] args) {
        try {
         Log4jAppender appender = new Log4jAppender();
            appender.setHostname("localhost");
            appender.setPort(41414);
            appender.setLayout(new PatternLayout("%d [%c] (%t) <%X{user}
%X{field}> %m"));
         //   appender.setReconnectAttempts(100);

           appender.activateOptions();

            log.addAppender(appender);

            MDC.put("user", "chris");
          //  while (true) {
                MDC.put("field", UUID.randomUUID().toString());
                log.info("=====> Hello World");
                try {
                    throw new Exception("Testing");
                } catch (Exception e) {
                    log.error("Gone wrong ===>", e);
                }
            //}
                System.in.read();
                System.in.read();
        }
        catch (Exception e) {
            e.printStackTrace();
        }
    }
}

I am receiving the only body messages as follows in HDFS.

*=====> Hello World *
*Gone wrong ===>*

I am missing any config here ?

khadar

On Sun, Jul 29, 2012 at 10:04 AM, Hari Shreedharan <
[EMAIL PROTECTED]> wrote:

>  + user@
>
> Thamatam,
>
> The Log4J appender adds the date, log level and logger name to the flume
> event headers and the text of the log event to the flume event body. The
> reason the log level and time are missing is that these are in the headers
> and the text serializer does not serialize the headers.
>
> To write to a file or HDFS, please use a Serializer together with the
> RollingFileSink or HDFSEventSink. Please take a look at the plain text
> serializer or Avro serializer to understand this better.
>
> Thanks,
> Hari
>
> --
> Hari Shreedharan
>
> On Saturday, July 28, 2012 at 5:47 PM, thamatam Jayaprakash wrote:
>
> Hi Hari,
>
>
> Actually im unable to send to this mail to the user and dev group so, im
> mailing to you .
>
> Could you pls point me where im going wrong.
Thanks,
Khadar

As I mentioned in the previous mail, you need to write a serializer to format the flume event to the format you want it to be written in. The text serializer does not write the headers out into the file. You can simply extend the Body text event serializer(org.apache.flume.serialization.BodyTextEventSerializer) to write the headers too.  

Thanks,
Hari

--
Hari Shreedharan
On Saturday, July 28, 2012 at 11:48 PM, khadar basha wrote:

> Hi  Hari,
>
> i am also facing same problem.  Here is my configuration.
>
>
> flume-conf.properties file
> =================> agent2Test1.sources = seqGenSrc
> agent2Test1.channels = memoryChannel
> agent2Test1.sinks = loggerSink
>
> # For each one of the sources, the type is defined
> agent2Test1.sources.seqGenSrc.type = avro
> agent2Test1.sources.seqGenSrc.bind=localhost
> agent2Test1.sources.seqGenSrc.port=41414
>
> # interceptors for host and date
> agent2Test1.sources.seqGenSrc.interceptors = time hostInterceptor
> agent2Test1.sources.seqGenSrc.interceptors.hostInterceptor.type = org.apache.flume.interceptor.HostInterceptor$Builder
> agent2Test1.sources.seqGenSrc.interceptors.hostInterceptor.hostHeader = host
> agent2Test1.sources.seqGenSrc.interceptors.hostInterceptor.useIP = false
> agent2Test1.sources.seqGenSrc.interceptors.hostInterceptor.hostHeader.preserveExisting = false
> agent2Test1.sources.seqGenSrc.interceptors.time.type = org.apache.flume.interceptor.TimestampInterceptor$Builder
>
> # The channel can be defined as follows.
> agent2Test1.sources.seqGenSrc.channels = memoryChannel
>
> # Each sink's type must be defined
> agent2Test1.sinks.loggerSink.type = hdfs
> agent2Test1.sinks.loggerSink.hdfs.path = hdfs://hadoopHost:8020/data/%Y/%m/%d/%{host}/Logs
>
> agent2Test1.sinks.loggerSink.hdfs.fileType = DataStream
>
> #Specify the channel the sink should use
> agent2Test1.sinks.loggerSink.channel = memoryChannel
>
> # Each channel's type is defined.
> agent2Test1.channels.memoryChannel.type = memory
>
> # Other config values specific to each type of channel(sink or source)
> # can be defined as well
> # In this case, it specifies the capacity of the memory channel
> agent2Test1.channels.memoryChannel.capacity = 1000
>
>
>
>
> Sample java program to generate the log message:
> ====================================>
>
> package com.test;
>
>
> import org.apache.flume.clients.log4jappender.Log4jAppender;
> import org.apache.log4j.Logger;
> import org.apache.log4j.MDC;
> import org.apache.log4j.PatternLayout;
>
> import java.util.UUID;
>
>
> public class Main {
>     static Logger log = Logger.getLogger(Main.class);
>
>     public static void main(String[] args) {
>         try {
>          Log4jAppender appender = new Log4jAppender();
>             appender.setHostname("localhost");
>             appender.setPort(41414);
>             appender.setLayout(new PatternLayout("%d [%c] (%t) <%X{user} %X{field}> %m"));
>          //   appender.setReconnectAttempts(100);
>            
>            appender.activateOptions();
>
>             log.addAppender(appender);
>
>             MDC.put("user", "chris");
>           //  while (true) {
>                 MDC.put("field", UUID.randomUUID().toString());
>                 log.info (http://log.info/)("=====> Hello World");
>                 try {
>                     throw new Exception("Testing");
>                 } catch (Exception e) {
>                     log.error("Gone wrong ===>", e);
>                 }
>             //}
>                 System.in.read();
>                 System.in.read();
>         }
>         catch (Exception e) {
>             e.printStackTrace();
>         }
>     }
> }
>
>
> I am receiving the only body messages as follows in HDFS.
>
> =====> Hello World
> Gone wrong ===>
>
> I am missing any config here ?
>
> khadar
>
> On Sun, Jul 29, 2012 at 10:04 AM, Hari Shreedharan <[EMAIL PROTECTED] (mailto:[EMAIL PROTECTED])> wrote:
> > + user@
> >
> > Thamatam,
>

Thanks Hari. I will try out that option.

-Khadar

On Sun, Jul 29, 2012 at 12:42 PM, Hari Shreedharan <
[EMAIL PROTECTED]> wrote:

>  As I mentioned in the previous mail, you need to write a serializer to
> format the flume event to the format you want it to be written in. The text
> serializer does not write the headers out into the file. You can simply
> extend the Body text event
> serializer(org.apache.flume.serialization.BodyTextEventSerializer) to write
> the headers too.
>
> Thanks,
> Hari
>
> --
> Hari Shreedharan
>
> On Saturday, July 28, 2012 at 11:48 PM, khadar basha wrote:
>
> Hi  Hari,
>
> i am also facing same problem.  Here is my configuration.
>
>
> flume-conf.properties file
> =================> agent2Test1.sources = seqGenSrc
> agent2Test1.channels = memoryChannel
> agent2Test1.sinks = loggerSink
>
> # For each one of the sources, the type is defined
> *agent2Test1.sources.seqGenSrc.type = avro*
> agent2Test1.sources.seqGenSrc.bind=localhost
> agent2Test1.sources.seqGenSrc.port=41414
>
> # interceptors for host and date
> agent2Test1.sources.seqGenSrc.interceptors = time hostInterceptor
> agent2Test1.sources.seqGenSrc.interceptors.hostInterceptor.type > org.apache.flume.interceptor.HostInterceptor$Builder
> agent2Test1.sources.seqGenSrc.interceptors.hostInterceptor.hostHeader > host
> agent2Test1.sources.seqGenSrc.interceptors.hostInterceptor.useIP = false
> agent2Test1.sources.seqGenSrc.interceptors.hostInterceptor.hostHeader.preserveExisting
> = false
> agent2Test1.sources.seqGenSrc.interceptors.time.type > org.apache.flume.interceptor.TimestampInterceptor$Builder
>
> # The channel can be defined as follows.
> agent2Test1.sources.seqGenSrc.channels = memoryChannel
>
> # Each sink's type must be defined
> agent2Test1.sinks.loggerSink.type = hdfs
> agent2Test1.sinks.loggerSink.hdfs.path > hdfs://hadoopHost:8020/data/%Y/%m/%d/%{host}/Logs
>
> agent2Test1.sinks.loggerSink.hdfs.fileType = DataStream
>
> #Specify the channel the sink should use
> agent2Test1.sinks.loggerSink.channel = memoryChannel
>
> # Each channel's type is defined.
> agent2Test1.channels.memoryChannel.type = memory
>
> # Other config values specific to each type of channel(sink or source)
> # can be defined as well
> # In this case, it specifies the capacity of the memory channel
> agent2Test1.channels.memoryChannel.capacity = 1000
>
>
>
> Sample java program to generate the log message:
> ====================================>
>
> package com.test;
>
>
> import org.apache.flume.clients.log4jappender.Log4jAppender;
> import org.apache.log4j.Logger;
> import org.apache.log4j.MDC;
> import org.apache.log4j.PatternLayout;
>
> import java.util.UUID;
>
>
> public class Main {
>     static Logger log = Logger.getLogger(Main.class);
>
>     public static void main(String[] args) {
>         try {
>          Log4jAppender appender = new Log4jAppender();
>             appender.setHostname("localhost");
>             appender.setPort(41414);
>             appender.setLayout(new PatternLayout("%d [%c] (%t) <%X{user}
> %X{field}> %m"));
>          //   appender.setReconnectAttempts(100);
>
>            appender.activateOptions();
>
>             log.addAppender(appender);
>
>             MDC.put("user", "chris");
>           //  while (true) {
>                 MDC.put("field", UUID.randomUUID().toString());
>                 log.info("=====> Hello World");
>                 try {
>                     throw new Exception("Testing");
>                 } catch (Exception e) {
>                     log.error("Gone wrong ===>", e);
>                 }
>             //}
>                 System.in.read();
>                 System.in.read();
>         }
>         catch (Exception e) {
>             e.printStackTrace();
>         }
>     }
> }
>
> I am receiving the only body messages as follows in HDFS.
>
> *=====> Hello World *
> *Gone wrong ===>*
>
> I am missing any config here ?
>
> khadar
>
> On Sun, Jul 29, 2012 at 10:04 AM, Hari Shreedharan <
> [EMAIL PROTECTED]> wrote:
Thanks,
Khadar

Thanks Hari ,

i got little progress.

But im getting garbage values.

this is my configurations:

*flume-conf.properties*
---------------------------------------
agent2.sources = seqGenSrc
agent2.channels = memoryChannel
agent2.sinks = loggerSink

agent2.sources.seqGenSrc.type = avro
agent2.sources.seqGenSrc.bind=localhost
agent2.sources.seqGenSrc.port=41414

agent2.channels.memoryChannel.type = memory
agent2.channels.memoryChannel.capacity = 1000000
agent2.channels.memoryChannel.transactionCapacity = 1000000
agent2.channels.memoryChannel.keep-alive = 30

agent2.sources.seqGenSrc.channels = memoryChannel

agent2.sinks.loggerSink.type = hdfs
agent2.sinks.loggerSink.hdfs.path = hdfs://ip:portno/data/CspcLogs
agent2.sinks.loggerSink.hdfs.fileType = DataStream
agent2.sinks.loggerSink.channel = memoryChannel
agent2.sinks.loggerSink.serializer = avro_event
agent2.sinks.loggerSink.serializer.compressionCodec = snappy
agent2.sinks.loggerSink.serializer.syncIntervalBytes = 2048000
agent2.channels.memoryChannel.type = memory
log4j.properties
------------------------------------------------------------------------------
log4j.rootLogger=INFO, CA, flume

log4j.appender.CA=org.apache.log4j.ConsoleAppender

log4j.appender.CA.layout=org.apache.log4j.PatternLayout
log4j.appender.CA.layout.ConversionPattern=%-4r [%t] %-5p %c %x - %m%n

log4j.appender.flume = org.apache.flume.clients.log4jappender.Log4jAppender
log4j.appender.flume.Hostname = localhost
log4j.appender.flume.Port = 41414
and my output:
------------------------
Obj#1;#4;#20;avro.codec#8;null#22;avro.schema�#2;{"type":"record","name":"Event","fields":[{"name":"headers","type":{"type":"map","values":"string"}},{"name":"body","type":"bytes"}]}�|#20;#27;��(r5��q
��nl#2;�#3;#8;8flume.client.log4j.log.level
40000Fflume.client.log4j.message.encoding#8;UTF88flume.client.log4j.timestamp#26;1343665387977<flume.client.log4j.logger.name2com.cisco.flume.FlumeTest�(Sample
error message|#20;#27;��(r5��q ����nl#2;�#3;#8;8flume.client.log4j.log.level
50000Fflume.client.log4j.message.encoding#8;UTF88flume.client.log4j.timestamp#26;1343665387993<flume.client.log4j.logger.name2com.cisco.flume.FlumeTest�(Sample
fatal message|#20;#27;��(r5��q ����nl#2;�#3;#8;8flume.client.log4j.log.level
20000Fflume.client.log4j.message.encoding#8;UTF88flume.client.log4j.timestamp#26;
Please let me know, if im in the wrong path.

Please suggest me to get custom logging pattern (for example like in log4j)
Thanks
JP

On Sun, Jul 29, 2012 at 10:04 AM, Hari Shreedharan <
[EMAIL PROTECTED]> wrote:

>  + user@
>
> Thamatam,
>
> The Log4J appender adds the date, log level and logger name to the flume
> event headers and the text of the log event to the flume event body. The
> reason the log level and time are missing is that these are in the headers
> and the text serializer does not serialize the headers.
>
> To write to a file or HDFS, please use a Serializer together with the
> RollingFileSink or HDFSEventSink. Please take a look at the plain text
> serializer or Avro serializer to understand this better.
>
> Thanks,
> Hari
>
> --
> Hari Shreedharan
>
> On Saturday, July 28, 2012 at 5:47 PM, thamatam Jayaprakash wrote:
>
> Hi Hari,
>
>
> Actually im unable to send to this mail to the user and dev group so, im
> mailing to you .
>
> Could you pls point me where im going wrong.
> *Please suggest me which log appender need to use for custom logging
> pattern and appender.*
>
> Im working on Flume 1.1.0 and 1.2.0 . We are not able set log pattern and
> We are using log4jappender
> log4j.appender.flume=org.apache.flume.clients.log4jappender.Log4jAppender
>
> but we are getting plain test
>
> *Example if i log following mssages :*
>
> 17:42:55,928  INFO SimpleJdbcServlet:69 - doGet of SimpleJdbcServlet
> ended...
> 17:43:03,489  INFO HelloServlet:29 - HelloServlet of doGet started...
> 17:43:03,489  INFO HelloServlet:33 -
>  Hello from Simple Servlet
> 17:43:03,489  INFO HelloServlet:35 - HelloServlet of doGet end...
JP

You are using the AvroEventSerializer. This formats the event into Avro format specified by org.apache.flume.serialization.FlumeEventAvroEventSerializer, which is why it looks like garbage, while it is not. Your app should be written to read and understand the Avro format. If you need it to human readable, you will need to write your own serializer, perhaps by extending the BodyTextEventSerializer.

Thanks
Hari

--  
Hari Shreedharan
On Monday, July 30, 2012 at 9:34 AM, JP wrote:

> Thanks Hari ,
>  
> i got little progress.
>  
> But im getting garbage values.
>  
> this is my configurations:
>  
> flume-conf.properties
> ---------------------------------------
> agent2.sources = seqGenSrc
> agent2.channels = memoryChannel
> agent2.sinks = loggerSink
>  
> agent2.sources.seqGenSrc.type = avro
> agent2.sources.seqGenSrc.bind=localhost
> agent2.sources.seqGenSrc.port=41414
>  
> agent2.channels.memoryChannel.type = memory
> agent2.channels.memoryChannel.capacity = 1000000
> agent2.channels.memoryChannel.transactionCapacity = 1000000
> agent2.channels.memoryChannel.keep-alive = 30
>  
> agent2.sources.seqGenSrc.channels = memoryChannel
>  
> agent2.sinks.loggerSink.type = hdfs
> agent2.sinks.loggerSink.hdfs.path = hdfs://ip:portno/data/CspcLogs
> agent2.sinks.loggerSink.hdfs.fileType = DataStream
> agent2.sinks.loggerSink.channel = memoryChannel
> agent2.sinks.loggerSink.serializer = avro_event
> agent2.sinks.loggerSink.serializer.compressionCodec = snappy
> agent2.sinks.loggerSink.serializer.syncIntervalBytes = 2048000
> agent2.channels.memoryChannel.type = memory
>  
>  
> log4j.properties
> ------------------------------------------------------------------------------
> log4j.rootLogger=INFO, CA, flume
>  
> log4j.appender.CA (http://log4j.appender.CA)=org.apache.log4j.ConsoleAppender
>  
> log4j.appender.CA.layout=org.apache.log4j.PatternLayout
> log4j.appender.CA.layout.ConversionPattern=%-4r [%t] %-5p %c %x - %m%n
>  
> log4j.appender.flume = org.apache.flume.clients.log4jappender.Log4jAppender
> log4j.appender.flume.Hostname = localhost
> log4j.appender.flume.Port = 41414
>  
>  
> and my output:
> ------------------------
> Objavro.codecnullavro.schema�{"type":"record","name":"Event","fields":[{"name":"headers","type":{"type":"map","values":"string"}},{"name":"body","type":"bytes"}]}�|��(r5���q��nl�8flume.client.log4j.log.level
> 40000Fflume.client.log4j.message.encodingUTF88flume.client.log4j.timestamp1343665387977<flume.client.log4j.logger.name2com.cisco.flume.FlumeTest�(Sample error message|��(r5���q��nl�8flume.client.log4j.log.level
> 50000Fflume.client.log4j.message.encodingUTF88flume.client.log4j.timestamp1343665387993<flume.client.log4j.logger.name2com.cisco.flume.FlumeTest�(Sample fatal message|��(r5���q��nl�8flume.client.log4j.log.level
> 20000Fflume.client.log4j.message.encodingUTF88flume.client.log4j.timestamp
>  
>  
> Please let me know, if im in the wrong path.
>  
> Please suggest me to get custom logging pattern (for example like in log4j)
>  
>  
> Thanks
> JP
>  
> On Sun, Jul 29, 2012 at 10:04 AM, Hari Shreedharan <[EMAIL PROTECTED] (mailto:[EMAIL PROTECTED])> wrote:
> > + user@  
> >  
> > Thamatam,
> >  
> > The Log4J appender adds the date, log level and logger name to the flume event headers and the text of the log event to the flume event body. The reason the log level and time are missing is that these are in the headers and the text serializer does not serialize the headers.  
> >  
> > To write to a file or HDFS, please use a Serializer together with the RollingFileSink or HDFSEventSink. Please take a look at the plain text serializer or Avro serializer to understand this better.  
> >  
> > Thanks,
> > Hari
> >  
> > --  
> > Hari Shreedharan
> >  
> >  
> > On Saturday, July 28, 2012 at 5:47 PM, thamatam Jayaprakash wrote:
> >  
> > > Hi Hari,
> > >  
> > >  
> > > Actually im unable to send to this mail to the user and dev group so, im mailing to you .  
> > >  
> > > Could you pls point me where im going wrong.

Check out the source code for the appender to see which headers you need to
write.

https://github.com/apache/flume/blob/trunk/flume-ng-clients/flume-ng-log4jappender/src/main/java/org/apache/flume/clients/log4jappender/Log4jAppender.java#L96

If you want to verify that the headers are being passed, try using a logger
sink in your Flume agent for debugging purposes.

If you want an example of writing an EventSerializer, I wrote up a little
bit of info here:
https://cwiki.apache.org/confluence/display/FLUME/Flume+1.x+Event+Serializers

Regards,
Mike
On Mon, Jul 30, 2012 at 11:22 AM, Hari Shreedharan <
[EMAIL PROTECTED]> wrote:

>  You are using the AvroEventSerializer. This formats the event into Avro
> format specified by org.apache.flume.serialization.FlumeEventAvroEventSerializer,
> which is why it looks like garbage, while it is not. Your app should be
> written to read and understand the Avro format. If you need it to human
> readable, you will need to write your own serializer, perhaps by extending
> the BodyTextEventSerializer.
>
> Thanks
> Hari
>
> --
> Hari Shreedharan
>
> On Monday, July 30, 2012 at 9:34 AM, JP wrote:
>
> Thanks Hari ,
>
> i got little progress.
>
> But im getting garbage values.
>
> this is my configurations:
>
> *flume-conf.properties*
> ---------------------------------------
> agent2.sources = seqGenSrc
> agent2.channels = memoryChannel
> agent2.sinks = loggerSink
>
> agent2.sources.seqGenSrc.type = avro
> agent2.sources.seqGenSrc.bind=localhost
> agent2.sources.seqGenSrc.port=41414
>
> agent2.channels.memoryChannel.type = memory
> agent2.channels.memoryChannel.capacity = 1000000
> agent2.channels.memoryChannel.transactionCapacity = 1000000
> agent2.channels.memoryChannel.keep-alive = 30
>
> agent2.sources.seqGenSrc.channels = memoryChannel
>
> agent2.sinks.loggerSink.type = hdfs
> agent2.sinks.loggerSink.hdfs.path = hdfs://ip:portno/data/CspcLogs
> agent2.sinks.loggerSink.hdfs.fileType = DataStream
> agent2.sinks.loggerSink.channel = memoryChannel
> agent2.sinks.loggerSink.serializer = avro_event
> agent2.sinks.loggerSink.serializer.compressionCodec = snappy
> agent2.sinks.loggerSink.serializer.syncIntervalBytes = 2048000
> agent2.channels.memoryChannel.type = memory
>
>
> log4j.properties
>
> ------------------------------------------------------------------------------
> log4j.rootLogger=INFO, CA, flume
>
> log4j.appender.CA=org.apache.log4j.ConsoleAppender
>
> log4j.appender.CA.layout=org.apache.log4j.PatternLayout
> log4j.appender.CA.layout.ConversionPattern=%-4r [%t] %-5p %c %x - %m%n
>
> log4j.appender.flume = org.apache.flume.clients.log4jappender.Log4jAppender
> log4j.appender.flume.Hostname = localhost
> log4j.appender.flume.Port = 41414
>
>
> and my output:
> ------------------------
> Obj avro.codec null avro.schema�
> {"type":"record","name":"Event","fields":[{"name":"headers","type":{"type":"map","values":"string"}},{"name":"body","type":"bytes"}]}�|
> ��(r5��q ��nl � 8flume.client.log4j.log.level
> 40000Fflume.client.log4j.message.encoding
> UTF88flume.client.log4j.timestamp
> 1343665387977<flume.client.log4j.logger.name2com.cisco.flume.FlumeTest���(Sample
> error message| ��(r5��q ���nl � 8flume.client.log4j.log.level
> 50000Fflume.client.log4j.message.encoding
> UTF88flume.client.log4j.timestamp
> 1343665387993<flume.client.log4j.logger.name2com.cisco.flume.FlumeTest���(Sample
> fatal message| ��(r5��q ���nl � 8flume.client.log4j.log.level
> 20000Fflume.client.log4j.message.encoding
> UTF88flume.client.log4j.timestamp
>
>
> Please let me know, if im in the wrong path.
>
> Please suggest me to get custom logging pattern (for example like in log4j)
>
>
> Thanks
> JP
>
> On Sun, Jul 29, 2012 at 10:04 AM, Hari Shreedharan <
> [EMAIL PROTECTED]> wrote:
>
>  + user@
>
> Thamatam,
>
> The Log4J appender adds the date, log level and logger name to the flume
> event headers and the text of the log event to the flume event body. The
> reason the log level and time are missing is that these are in the headers

Thanks Mike,

Unfortunately the following URL's are not working:

https://svn.apache.org/viewvc/incubator/flume/trunk/flume-ng-core/src/test/java/org/apache/flume/serialization/SyslogAvroEventSerializer.java?view=markup

https://svn.apache.org/viewvc/incubator/flume/trunk/flume-ng-core/src/test/java/org/apache/flume/serialization/

Thanks
JP
On Tue, Jul 31, 2012 at 4:03 AM, Mike Percy <[EMAIL PROTECTED]> wrote:

> Check out the source code for the appender to see which headers you need
> to write.
>
>
> https://github.com/apache/flume/blob/trunk/flume-ng-clients/flume-ng-log4jappender/src/main/java/org/apache/flume/clients/log4jappender/Log4jAppender.java#L96
>
> If you want to verify that the headers are being passed, try using a
> logger sink in your Flume agent for debugging purposes.
>
> If you want an example of writing an EventSerializer, I wrote up a little
> bit of info here:
> https://cwiki.apache.org/confluence/display/FLUME/Flume+1.x+Event+Serializers
>
> Regards,
> Mike
>
>
>
> On Mon, Jul 30, 2012 at 11:22 AM, Hari Shreedharan <
> [EMAIL PROTECTED]> wrote:
>
>>  You are using the AvroEventSerializer. This formats the event into Avro
>> format specified by org.apache.flume.serialization.FlumeEventAvroEventSerializer,
>> which is why it looks like garbage, while it is not. Your app should be
>> written to read and understand the Avro format. If you need it to human
>> readable, you will need to write your own serializer, perhaps by extending
>> the BodyTextEventSerializer.
>>
>> Thanks
>> Hari
>>
>> --
>> Hari Shreedharan
>>
>> On Monday, July 30, 2012 at 9:34 AM, JP wrote:
>>
>> Thanks Hari ,
>>
>> i got little progress.
>>
>> But im getting garbage values.
>>
>> this is my configurations:
>>
>> *flume-conf.properties*
>> ---------------------------------------
>> agent2.sources = seqGenSrc
>> agent2.channels = memoryChannel
>> agent2.sinks = loggerSink
>>
>> agent2.sources.seqGenSrc.type = avro
>> agent2.sources.seqGenSrc.bind=localhost
>> agent2.sources.seqGenSrc.port=41414
>>
>> agent2.channels.memoryChannel.type = memory
>> agent2.channels.memoryChannel.capacity = 1000000
>> agent2.channels.memoryChannel.transactionCapacity = 1000000
>> agent2.channels.memoryChannel.keep-alive = 30
>>
>> agent2.sources.seqGenSrc.channels = memoryChannel
>>
>> agent2.sinks.loggerSink.type = hdfs
>> agent2.sinks.loggerSink.hdfs.path = hdfs://ip:portno/data/CspcLogs
>> agent2.sinks.loggerSink.hdfs.fileType = DataStream
>> agent2.sinks.loggerSink.channel = memoryChannel
>> agent2.sinks.loggerSink.serializer = avro_event
>> agent2.sinks.loggerSink.serializer.compressionCodec = snappy
>> agent2.sinks.loggerSink.serializer.syncIntervalBytes = 2048000
>> agent2.channels.memoryChannel.type = memory
>>
>>
>> log4j.properties
>>
>> ------------------------------------------------------------------------------
>> log4j.rootLogger=INFO, CA, flume
>>
>> log4j.appender.CA=org.apache.log4j.ConsoleAppender
>>
>> log4j.appender.CA.layout=org.apache.log4j.PatternLayout
>> log4j.appender.CA.layout.ConversionPattern=%-4r [%t] %-5p %c %x - %m%n
>>
>> log4j.appender.flume >> org.apache.flume.clients.log4jappender.Log4jAppender
>> log4j.appender.flume.Hostname = localhost
>> log4j.appender.flume.Port = 41414
>>
>>
>> and my output:
>> ------------------------
>> Obj avro.codec null avro.schema�
>> {"type":"record","name":"Event","fields":[{"name":"headers","type":{"type":"map","values":"string"}},{"name":"body","type":"bytes"}]}�|
>> ��(r5��q ��nl � 8flume.client.log4j.log.level
>> 40000Fflume.client.log4j.message.encoding
>> UTF88flume.client.log4j.timestamp
>> 1343665387977<flume.client.log4j.logger.name2com.cisco.flume.FlumeTest�(Sample
>> error message| ��(r5��q ���nl � 8flume.client.log4j.log.level
>> 50000Fflume.client.log4j.message.encoding
>> UTF88flume.client.log4j.timestamp
>> 1343665387993<flume.client.log4j.logger.name2com.cisco.flume.FlumeTest�(Sample
>> fatal message| ��(r5��q ���nl � 8flume.client.log4j.log.level
>> 20000Fflume.client.log4j.message.encoding
JP