Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Flume >> mail # user >> Memory Channel Error on Load Testing


Copy link to this message
-
Memory Channel Error on Load Testing
Hi
I am getting error while load testing a flume agent with HTTSrc and Avro
Sink.

Is there an optimum configuration for memory channel for get flume working
on high load. Thanks in advance for any inputs.
tier1.sources  = httpsrc1
tier1.channels = c1   c2
tier1.sinks    = avrosink1

tier1.sources.httpsrc1.bind     = host1
tier1.sources.httpsrc1.type = http
tier1.sources.httpsrc1.port = 443
tier1.sources.httpsrc1.channels = c1 c2
tier1.sources.httpsrc1.handler =flume.SimpleHTTPEventHandler
tier1.sources.httpsrc1.handler.nickname = HTTPTesting

tier1.channels.c1.type   = memory
tier1.channels.c1.capacity = 10000
tier1.channels.c1.transactionCapacity = 1000
tier1.sinks.sink1.channel      = c1
tier1.sinks.avrosink1.channel      = c1

tier1.channels.c2.type   = memory
tier1.channels.c2.capacity = 100000
tier1.channels.c2.transactionCapacity = 10000

#tier1.channels.c1.type   = file
#tier1.channels.c1.checkpointDir = /tmp/flume/checkpoint
#tier1.channels.c1.dataDirs = /tmp/flume/data
#tier1.sinks.avrosink1.channel      = c1
tier1.sinks.avrosink1.type = avro
tier1.sinks.avrosink1.hostname = ipaddr
tier1.sinks.avrosink1.port = 4444
tier1.sinks.avrosink1.avro.fileType = DataStream
tier1.sinks.avrosink1.serializer = avro_event
tier1.sinks.avrosink1.serializer.compressionCodec = snappy

tier1.sinks = filesink1
tier1.sinks.filesink1.type = file_roll
tier1.sinks.filesink1.channel = c2
tier1.sinks.filesink1.batchSize = 10000
#tier1.sinks.filesink1.rollInterval = 86400
tier1.sinks.filesink1.rollInterval =0
tier1.sinks.filesink1.sink.directory = /projects/logs

WARN http.HTTPSource: Error appending event to channel. Channel might be
full. Consider increasing the channel capacity or make sure the sinks
perform faster.
org.apache.flume.ChannelException: Unable to put batch on required channel:
org.apache.flume.channel.MemoryChannel{name: c1}
at
org.apache.flume.channel.ChannelProcessor.processEventBatch(ChannelProcessor.java:200)
at
org.apache.flume.source.http.HTTPSource$FlumeHTTPServlet.doPost(HTTPSource.java:201)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:725)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:814)
at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:401)
at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
at org.mortbay.jetty.Server.handle(Server.java:326)
at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
at
org.mortbay.jetty.HttpConnection$RequestHandler.content(HttpConnection.java:945)
at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:756)
at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:218)
at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
at
org.mortbay.jetty.bio.SocketConnector$Connection.run(SocketConnector.java:228)
at
org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
Caused by: org.apache.flume.ChannelException: Space for commit to queue
couldn't be acquired Sinks are likely not keeping up with sources, or the
buffer size is too tight
at
org.apache.flume.channel.MemoryChannel$MemoryTransaction.doCommit(MemoryChannel.java:128)
at
org.apache.flume.channel.BasicTransactionSemantics.commit(BasicTransactionSemantics.java:151)
at
org.apache.flume.channel.ChannelProcessor.processEventBatch(ChannelProcessor.java:192)
--
Deepak Subhramanian
+
Roshan Naik 2013-10-18, 22:55
+
Deepak Subhramanian 2013-10-23, 15:17
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB