Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Flume >> mail # user >> Flume not moving data to HDFS or local


+
Siddharth Tiwari 2013-10-31, 18:52
Copy link to this message
-
RE: Flume not moving data to HDFS or local
Try bumping your memory channel capacities up, they are the same as the batch size. I would go to at least 1000 on each mem channel.

Also, what to the logs and metrics show?

From: Siddharth Tiwari [mailto:[EMAIL PROTECTED]]
Sent: Thursday, October 31, 2013 11:53 AM
To: [EMAIL PROTECTED]
Subject: Flume not moving data to HDFS or local

Hi team I created flume source and sink as following in hadoop yarn and I am not getting data transferred from source to sink in HDFS it doesnt create any file and on local everytime I start agent it creates one empty file. Below are my configs in source and sink
Source :-
agent.sources = logger1
agent.sources.logger1.type = exec
agent.sources.logger1.command = tail -f /var/log/messages
agent.sources.logger1.batchsSize = 0
agent.sources.logger1.channels = memoryChannel
agent.channels = memoryChannel
agent.channels.memoryChannel.type = memory
agent.channels.memoryChannel.capacity = 100
agent.sinks = AvroSink
agent.sinks.AvroSink.type = avro
agent.sinks.AvroSink.channel = memoryChannel
agent.sinks.AvroSink.hostname = 192.168.147.101
agent.sinks.AvroSink.port = 4545
agent.sources.logger1.interceptors = itime ihost
agent.sources.logger1.interceptors.itime.type = TimestampInterceptor
agent.sources.logger1.interceptors.ihost.type = host
agent.sources.logger1.interceptors.ihost.useIP = false
agent.sources.logger1.interceptors.ihost.hostHeader = host
Sink at one of the slave ( datanodes on my Yarn cluster ) :

collector.sources = AvroIn
collector.sources.AvroIn.type = avro
collector.sources.AvroIn.bind = 0.0.0.0
collector.sources.AvroIn.port = 4545
collector.sources.AvroIn.channels = mc1 mc2
collector.channels = mc1 mc2
collector.channels.mc1.type = memory
collector.channels.mc1.capacity = 100

collector.channels.mc2.type = memory
collector.channels.mc2.capacity = 100

collector.sinks = LocalOut HadoopOut
collector.sinks.LocalOut.type = file_roll
collector.sinks.LocalOut.sink.directory = /home/hadoop/flume
collector.sinks.LocalOut.sink.rollInterval = 0
collector.sinks.LocalOut.channel = mc1
collector.sinks.HadoopOut.type = hdfs
collector.sinks.HadoopOut.channel = mc2
collector.sinks.HadoopOut.hdfs.path = /flume
collector.sinks.HadoopOut.hdfs.fileType = DataStream
collector.sinks.HadoopOut.hdfs.writeFormat = Text
collector.sinks.HadoopOut.hdfs.rollSize = 0
collector.sinks.HadoopOut.hdfs.rollCount = 10000
collector.sinks.HadoopOut.hdfs.rollInterval = 600
can somebody point me to what am I doing wrong ?

This is what I get in my local directory

[hadoop@node1 flume]$ ls -lrt
total 0
-rw-rw-r-- 1 hadoop hadoop 0 Oct 31 11:25 1383243942803-1
-rw-rw-r-- 1 hadoop hadoop 0 Oct 31 11:28 1383244097923-1
-rw-rw-r-- 1 hadoop hadoop 0 Oct 31 11:31 1383244302225-1
-rw-rw-r-- 1 hadoop hadoop 0 Oct 31 11:33 1383244404929-1
when I restart the collector it creates one 0 bytes file.

Please help
*------------------------*
Cheers !!!
Siddharth Tiwari
Have a refreshing day !!!
"Every duty is holy, and devotion to duty is the highest form of worship of God."
"Maybe other people will try to limit me but I don't limit myself"
+
Siddharth Tiwari 2013-10-31, 19:29
+
Siddharth Tiwari 2013-10-31, 19:46
+
Paul Chavez 2013-10-31, 21:38
+
Siddharth Tiwari 2013-11-01, 02:05
+
Paul Chavez 2013-11-01, 03:10
+
Siddharth Tiwari 2013-11-01, 06:17
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB