Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Flume, mail # user - Log Events get Lost - flume 1.3


Copy link to this message
-
Log Events get Lost - flume 1.3
Kumar, Deepak8 2013-04-16, 08:16
Hi,

I have 10 flume agents configured at a single machine. A single log file has frequency of 500 log events/sec. Hence in 10 log files the logs are getting generated as 5000 log events per second (5000/sec).

If my channel capacity is 1 million,  more than 70% of log events is lost! If I increase the channel capacity to 50 millions, then flume agent takes more than 24 hours to transfer the log events from source to sink.

The size of dataDir (agent.channels.fileChannel.dataDirs = /var/log/flume-ng/file-channel/data) is almost 2G all the time.

Could you please guide me the optimum configuration so that I don't miss any of log events & the transfer is also good enough. My flume-conf.properties has following contents:

agent.channels = fileChannel

agent.sinks = avroSink

# Each sink's type must be defined

agent.sinks.avroSink.type = avro

agent.sinks.avroSink.hostname = spnnq01.nam.nsroot.net

agent.sinks.avroSink.port = 1442

agent.sinks.avroSink.batchSize = 1000

#Specify the channel the sink should use

agent.sinks.avroSink.channel = fileChannel

# Each channel's type is defined.

agent.channels.fileChannel.type = file

agent.channels.fileChannel.checkpointDir = /var/log/flume-ng/file-channel/checkpoint

agent.channels.fileChannel.dataDirs = /var/log/flume-ng/file-channel/data

agent.channels.fileChannel.transactionCapacity = 1000

agent.channels.fileChannel.checkpointInterval = 30000

agent.channels.fileChannel.maxFileSize = 2146435071

agent.channels.fileChannel.minimumRequiredSpace = 524288000

agent.channels.fileChannel.keep-alive = 5

agent.channels.fileChannel.write-timeout = 10

agent.channels.fileChannel.checkpoint-timeout = 600

agent.channels.fileChannel.capacity = 50000000

agent.sources.s2.batchSize = 1

agent.sources.s2.channels = fileChannel

agent.sources.s2.command = tail -F /var/log/creditcard/AggKeyListener.2.2013-01-19

agent.sources.s2.interceptors = logIntercept

agent.sources.s2.interceptors.logIntercept.appId = 153299

agent.sources.s2.interceptors.logIntercept.env = SP

agent.sources.s2.interceptors.logIntercept.hostName = vm-e61b-fe34.nam.nsroot.net

agent.sources.s2.interceptors.logIntercept.logFileName = AggKeyListener.2.2013-01-19

agent.sources.s2.interceptors.logIntercept.logFilePath = /var/log/creditcard/

agent.sources.s2.interceptors.logIntercept.logType = creditcard log

agent.sources.s2.interceptors.logIntercept.type = com.citi.sponge.flume.agent.source.LogInterceptor$Builder

agent.sources.s2.type = exec

agent.sources.s0.batchSize = 1

agent.sources.s0.channels = fileChannel

agent.sources.s0.command = tail -F /var/log/creditcard/AggKeyListener.0.2013-01-19

agent.sources.s0.interceptors = logIntercept

agent.sources.s0.interceptors.logIntercept.appId = 153299

agent.sources.s0.interceptors.logIntercept.env = SP

agent.sources.s0.interceptors.logIntercept.hostName = vm-e61b-fe34.nam.nsroot.net

agent.sources.s0.interceptors.logIntercept.logFileName = AggKeyListener.0.2013-01-19

agent.sources.s0.interceptors.logIntercept.logFilePath = /var/log/creditcard/

agent.sources.s0.interceptors.logIntercept.logType = creditcard log

agent.sources.s0.interceptors.logIntercept.type = com.citi.sponge.flume.agent.source.LogInterceptor$Builder

agent.sources.s0.type = exec

agent.sources.s1.batchSize = 1

agent.sources.s1.channels = fileChannel

agent.sources.s1.command = tail -F /var/log/creditcard/AggKeyListener.1.2013-01-19

agent.sources.s1.interceptors = logIntercept

agent.sources.s1.interceptors.logIntercept.appId = 153299

agent.sources.s1.interceptors.logIntercept.env = SP

agent.sources.s1.interceptors.logIntercept.hostName = vm-e61b-fe34.nam.nsroot.net

agent.sources.s1.interceptors.logIntercept.logFileName = AggKeyListener.1.2013-01-19

agent.sources.s1.interceptors.logIntercept.logFilePath = /var/log/creditcard/

agent.sources.s1.interceptors.logIntercept.logType = creditcard log

agent.sources.s1.interceptors.logIntercept.type = com.citi.sponge.flume.agent.source.LogInterceptor$Builder

agent.sources.s1.type = exec

agent.sources.s3.batchSize = 1

agent.sources.s3.channels = fileChannel

agent.sources.s3.command = tail -F /var/log/creditcard/AggKeyListener.3.2013-01-19

agent.sources.s3.interceptors = logIntercept

agent.sources.s3.interceptors.logIntercept.appId = 153299

agent.sources.s3.interceptors.logIntercept.env = SP

agent.sources.s3.interceptors.logIntercept.hostName = vm-e61b-fe34.nam.nsroot.net

agent.sources.s3.interceptors.logIntercept.logFileName = AggKeyListener.3.2013-01-19

agent.sources.s3.interceptors.logIntercept.logFilePath = /var/log/creditcard/

agent.sources.s3.interceptors.logIntercept.logType = creditcard log

agent.sources.s3.interceptors.logIntercept.type = com.citi.sponge.flume.agent.source.LogInterceptor$Builder

agent.sources.s3.type = exec

agent.sources.s4.batchSize = 1

agent.sources.s4.channels = fileChannel

agent.sources.s4.command = tail -F /var/log/creditcard/AggKeyListener.4.2013-01-19

agent.sources.s4.interceptors = logIntercept

agent.sources.s4.interceptors.logIntercept.appId = 153299

agent.sources.s4.interceptors.logIntercept.env = SP

agent.sources.s4.interceptors.logIntercept.hostName = vm-e61b-fe34.nam.nsroot.net

agent.sources.s4.interceptors.logIntercept.logFileName = AggKeyListener.4.2013-01-19

agent.sources.s4.interceptors.logIntercept.logFilePath = /var/log/creditcard/

agent.sources.s4.interceptors.logIntercept.logType = creditcard log

agent.sources.s4.interceptors.logIntercept.type = com.citi.sponge.flume.agent.source.LogInterceptor$Builder

agent.sources.s4.type = exec

agent.sources.s5.batchSize = 1

agent.sources.s5.channels = fileChannel

agent.sources.s5.command = tail -F /var/log/creditcard/AggKeyListener.5.2013-01-19

agent.sources.s5.interceptors = logIntercept

agent.sources.s5.interceptors.logIntercept.appId = 153299

agent.sources.s5.interceptors.logIntercept.env = SP

agent.sources.s5.interceptors.logIntercept.hostName = vm-e61b-fe34.nam.nsroot.net

agent.so