Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Flume, mail # user - flume to HDFS log event write


+
Chhaya Vishwakarma 2013-01-09, 10:02
Copy link to this message
-
Re: flume to HDFS log event write
Bertrand Dechoux 2013-01-09, 10:09
http://hadoop.apache.org/docs/current/api/org/apache/hadoop/io/SequenceFile.html

is a binary format. You may want to make flume ouput to a file or the
console first.
And then compare what you are expecting versus what you are getting.

Regards

Bertrand

On Wed, Jan 9, 2013 at 11:02 AM, Chhaya Vishwakarma <
[EMAIL PROTECTED]> wrote:

>  hi,****
>
> ** **
>
> I am using Flume log4j appender to write log events to HDFS but it
> contains some junk value and I am not able to see anything other than log
> message no timestamp.****
>
> ** **
>
> Here is my configuration****
>
> *Log4j.properties*
>
> ** **
>
> log4j.logger.log4jExample= DEBUG,out2****
>
> log4j.appender.out2 = org.apache.flume.clients.log4jappender.Log4jAppender
> ****
>
> log4j.appender.out2.Port = 41414****
>
> log4j.appender.out2.Hostname = 172.20.104.223****
>
> ** **
>
> *here is agent configuration*
>
> a1.sources = r1****
>
> a1.sinks = k1****
>
> a1.channels = c1****
>
> ** **
>
> #sources****
>
> a1.sources.r1.type = avro****
>
> a1.sources.r1.bind =172.20.104.226****
>
> a1.sources.r1.port= 41414****
>
> a1.sources.r1.restart =true****
>
> a1.sources.r1.batchsize=10000****
>
> ** **
>
> # Describe the sink****
>
> a1.sinks.k1.type = hdfs****
>
> a1.sinks.k1.hdfs.path=hdfs://172.20.104.226:8020/flumeinput/%{host}****
>
> a1.sinks.k1.hdfs.file.Type=DataStream****
>
> a1.sinks.k1.hdfs.writeFormat=Writable****
>
> a1.sinks.k1.hdfs.rollCount=10000****
>
> a1.sinks.k1.serializer=TEXT****
>
> ** **
>
> # Use a channel which buffers events in memory****
>
> a1.channels.c1.type = file****
>
> a1.channels.c1.capacity = 10000****
>
> a1.channels.c1.transactionCapacity = 10000****
>
> ** **
>
> # Bind the source and sink to the channel****
>
> a1.sources.r1.channels = c1****
>
> a1.sinks.k1.channel = c1****
>
> * *
>
> *Expected output*
>
> [2013-01-09 15:15:45,457] - [main] DEBUG log4jExample Current data
> unavailalbe, using cached values****
>
> [2013-01-09 15:15:45,458] - [main] INFO  log4jExample Hello this is an
> info message****
>
> [2013-01-09 15:15:45,460] - [main] ERROR log4jExample Dabase unavaliable,
> connetion lost****
>
> [2013-01-09 15:15:45,461] - [main] WARN  log4jExample Attention!!
> Application running in debugmode****
>
> [2013-01-09 15:15:45,463] - [main] DEBUG log4jExample Current data
> unavailalbe, using cached values****
>
> [2013-01-09 15:15:45,465] - [main] INFO  log4jExample Hello this is an
> info message****
>
> [2013-01-09 15:15:45,467] - [main] ERROR log4jExample Dabase unavaliable,
> connetion lost****
>
> [2013-01-09 15:15:45,468] - [main] WARN  log4jExample Attention!!
> Application running in debugmode****
>
> [2013-01-09 15:15:45,470] - [main] DEBUG log4jExample Current data
> unavailalbe, using cached values****
>
> * *
>
> But getting this ****
>
> *Output on HDFS*
>
> SEQ#6;!org.apache.hadoop.io.LongWritable"org.apache.hadoop.io.BytesWritable
> ������+�#19;AE����9#8;<‑��-Current data unavailalbe, using cached values)#8;<‑��Hello
> this is an info message.#8;<‑��"Dabase unavaliable, connetion lost8#8;<‑��,Attention!!
> Application running in debugmode9#8;<‑��-Current data unavailalbe, using
> cached values)#8;<‑��Hello this is an info message.#8;<‑��"Dabase
> unavaliable, connetion lost8#8;<‑��#28;,Attention!! Application running in
> debugmode9#8;<‑��-Current data unavailalbe, using cached values)#8;<‑��‑Hello
> this is an info message.#8;<‑��‑"Dabase unavaliable, connetion lost8#8;<‑��­,Attention!!
> Application running in debugmode9#8;<‑�� -Current data unavailalbe, using
> cached values)#8;<‑�� Hello this is an info message.#8;<‑��!"Dabase
> unavaliable, connetion lost8#8;<‑��",Attention!! Application running in
> debugmode9#8;<‑��"-Current data unavailalbe, using cached values)#8;<‑��#Hello
> this is an info message.#8;<‑��#"Dabase unavaliable, connetion lost8#8;<‑��$,Attention!!
> Application running in debugmode9#8;<‑��$-Current data unavailalbe, using
> cached values)#8;<‑��%Hello this is an info message.#8;<‑��%"Dabase

Bertrand Dechoux
+
Chhaya Vishwakarma 2013-01-09, 10:12
+
Hari Shreedharan 2013-01-09, 17:07
+
Chhaya Vishwakarma 2013-01-09, 10:29