Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Flume, mail # user - Flume not moving data to HDFS or local


+
Siddharth Tiwari 2013-10-31, 18:52
Copy link to this message
-
RE: Flume not moving data to HDFS or local
Paul Chavez 2013-10-31, 19:19
Try bumping your memory channel capacities up, they are the same as the batch size. I would go to at least 1000 on each mem channel.

Also, what to the logs and metrics show?

From: Siddharth Tiwari [mailto:[EMAIL PROTECTED]]
Sent: Thursday, October 31, 2013 11:53 AM
To: [EMAIL PROTECTED]
Subject: Flume not moving data to HDFS or local

Hi team I created flume source and sink as following in hadoop yarn and I am not getting data transferred from source to sink in HDFS it doesnt create any file and on local everytime I start agent it creates one empty file. Below are my configs in source and sink
Source :-
agent.sources = logger1
agent.sources.logger1.type = exec
agent.sources.logger1.command = tail -f /var/log/messages
agent.sources.logger1.batchsSize = 0
agent.sources.logger1.channels = memoryChannel
agent.channels = memoryChannel
agent.channels.memoryChannel.type = memory
agent.channels.memoryChannel.capacity = 100
agent.sinks = AvroSink
agent.sinks.AvroSink.type = avro
agent.sinks.AvroSink.channel = memoryChannel
agent.sinks.AvroSink.hostname = 192.168.147.101
agent.sinks.AvroSink.port = 4545
agent.sources.logger1.interceptors = itime ihost
agent.sources.logger1.interceptors.itime.type = TimestampInterceptor
agent.sources.logger1.interceptors.ihost.type = host
agent.sources.logger1.interceptors.ihost.useIP = false
agent.sources.logger1.interceptors.ihost.hostHeader = host
Sink at one of the slave ( datanodes on my Yarn cluster ) :

collector.sources = AvroIn
collector.sources.AvroIn.type = avro
collector.sources.AvroIn.bind = 0.0.0.0
collector.sources.AvroIn.port = 4545
collector.sources.AvroIn.channels = mc1 mc2
collector.channels = mc1 mc2
collector.channels.mc1.type = memory
collector.channels.mc1.capacity = 100

collector.channels.mc2.type = memory
collector.channels.mc2.capacity = 100

collector.sinks = LocalOut HadoopOut
collector.sinks.LocalOut.type = file_roll
collector.sinks.LocalOut.sink.directory = /home/hadoop/flume
collector.sinks.LocalOut.sink.rollInterval = 0
collector.sinks.LocalOut.channel = mc1
collector.sinks.HadoopOut.type = hdfs
collector.sinks.HadoopOut.channel = mc2
collector.sinks.HadoopOut.hdfs.path = /flume
collector.sinks.HadoopOut.hdfs.fileType = DataStream
collector.sinks.HadoopOut.hdfs.writeFormat = Text
collector.sinks.HadoopOut.hdfs.rollSize = 0
collector.sinks.HadoopOut.hdfs.rollCount = 10000
collector.sinks.HadoopOut.hdfs.rollInterval = 600
can somebody point me to what am I doing wrong ?

This is what I get in my local directory

[hadoop@node1 flume]$ ls -lrt
total 0
-rw-rw-r-- 1 hadoop hadoop 0 Oct 31 11:25 1383243942803-1
-rw-rw-r-- 1 hadoop hadoop 0 Oct 31 11:28 1383244097923-1
-rw-rw-r-- 1 hadoop hadoop 0 Oct 31 11:31 1383244302225-1
-rw-rw-r-- 1 hadoop hadoop 0 Oct 31 11:33 1383244404929-1
when I restart the collector it creates one 0 bytes file.

Please help
*------------------------*
Cheers !!!
Siddharth Tiwari
Have a refreshing day !!!
"Every duty is holy, and devotion to duty is the highest form of worship of God."
"Maybe other people will try to limit me but I don't limit myself"
+
Siddharth Tiwari 2013-10-31, 19:29
+
Siddharth Tiwari 2013-10-31, 19:46
+
Paul Chavez 2013-10-31, 21:38
+
Siddharth Tiwari 2013-11-01, 02:05
+
Paul Chavez 2013-11-01, 03:10
+
Siddharth Tiwari 2013-11-01, 06:17