Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Flume >> mail # user >> Re: regarding flume


Copy link to this message
-
Re: regarding flume
Check out the source code for the appender to see which headers you need to
write.

https://github.com/apache/flume/blob/trunk/flume-ng-clients/flume-ng-log4jappender/src/main/java/org/apache/flume/clients/log4jappender/Log4jAppender.java#L96

If you want to verify that the headers are being passed, try using a logger
sink in your Flume agent for debugging purposes.

If you want an example of writing an EventSerializer, I wrote up a little
bit of info here:
https://cwiki.apache.org/confluence/display/FLUME/Flume+1.x+Event+Serializers

Regards,
Mike
On Mon, Jul 30, 2012 at 11:22 AM, Hari Shreedharan <
[EMAIL PROTECTED]> wrote:

>  You are using the AvroEventSerializer. This formats the event into Avro
> format specified by org.apache.flume.serialization.FlumeEventAvroEventSerializer,
> which is why it looks like garbage, while it is not. Your app should be
> written to read and understand the Avro format. If you need it to human
> readable, you will need to write your own serializer, perhaps by extending
> the BodyTextEventSerializer.
>
> Thanks
> Hari
>
> --
> Hari Shreedharan
>
> On Monday, July 30, 2012 at 9:34 AM, JP wrote:
>
> Thanks Hari ,
>
> i got little progress.
>
> But im getting garbage values.
>
> this is my configurations:
>
> *flume-conf.properties*
> ---------------------------------------
> agent2.sources = seqGenSrc
> agent2.channels = memoryChannel
> agent2.sinks = loggerSink
>
> agent2.sources.seqGenSrc.type = avro
> agent2.sources.seqGenSrc.bind=localhost
> agent2.sources.seqGenSrc.port=41414
>
> agent2.channels.memoryChannel.type = memory
> agent2.channels.memoryChannel.capacity = 1000000
> agent2.channels.memoryChannel.transactionCapacity = 1000000
> agent2.channels.memoryChannel.keep-alive = 30
>
> agent2.sources.seqGenSrc.channels = memoryChannel
>
> agent2.sinks.loggerSink.type = hdfs
> agent2.sinks.loggerSink.hdfs.path = hdfs://ip:portno/data/CspcLogs
> agent2.sinks.loggerSink.hdfs.fileType = DataStream
> agent2.sinks.loggerSink.channel = memoryChannel
> agent2.sinks.loggerSink.serializer = avro_event
> agent2.sinks.loggerSink.serializer.compressionCodec = snappy
> agent2.sinks.loggerSink.serializer.syncIntervalBytes = 2048000
> agent2.channels.memoryChannel.type = memory
>
>
> log4j.properties
>
> ------------------------------------------------------------------------------
> log4j.rootLogger=INFO, CA, flume
>
> log4j.appender.CA=org.apache.log4j.ConsoleAppender
>
> log4j.appender.CA.layout=org.apache.log4j.PatternLayout
> log4j.appender.CA.layout.ConversionPattern=%-4r [%t] %-5p %c %x - %m%n
>
> log4j.appender.flume = org.apache.flume.clients.log4jappender.Log4jAppender
> log4j.appender.flume.Hostname = localhost
> log4j.appender.flume.Port = 41414
>
>
> and my output:
> ------------------------
> Obj avro.codec null avro.schema�
> {"type":"record","name":"Event","fields":[{"name":"headers","type":{"type":"map","values":"string"}},{"name":"body","type":"bytes"}]}�|
> ��(r5��q ��nl � 8flume.client.log4j.log.level
> 40000Fflume.client.log4j.message.encoding
> UTF88flume.client.log4j.timestamp
> 1343665387977<flume.client.log4j.logger.name2com.cisco.flume.FlumeTest�(Sample
> error message| ��(r5��q ��nl � 8flume.client.log4j.log.level
> 50000Fflume.client.log4j.message.encoding
> UTF88flume.client.log4j.timestamp
> 1343665387993<flume.client.log4j.logger.name2com.cisco.flume.FlumeTest�(Sample
> fatal message| ��(r5��q ��nl � 8flume.client.log4j.log.level
> 20000Fflume.client.log4j.message.encoding
> UTF88flume.client.log4j.timestamp
>
>
> Please let me know, if im in the wrong path.
>
> Please suggest me to get custom logging pattern (for example like in log4j)
>
>
> Thanks
> JP
>
> On Sun, Jul 29, 2012 at 10:04 AM, Hari Shreedharan <
> [EMAIL PROTECTED]> wrote:
>
>  + user@
>
> Thamatam,
>
> The Log4J appender adds the date, log level and logger name to the flume
> event headers and the text of the log event to the flume event body. The
> reason the log level and time are missing is that these are in the headers
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB