Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Flume >> mail # user >> Memory Channel Error on Load Testing


Copy link to this message
-
Memory Channel Error on Load Testing
Hi
I am getting error while load testing a flume agent with HTTSrc and Avro
Sink.

Is there an optimum configuration for memory channel for get flume working
on high load. Thanks in advance for any inputs.
tier1.sources  = httpsrc1
tier1.channels = c1   c2
tier1.sinks    = avrosink1

tier1.sources.httpsrc1.bind     = host1
tier1.sources.httpsrc1.type = http
tier1.sources.httpsrc1.port = 443
tier1.sources.httpsrc1.channels = c1 c2
tier1.sources.httpsrc1.handler =flume.SimpleHTTPEventHandler
tier1.sources.httpsrc1.handler.nickname = HTTPTesting

tier1.channels.c1.type   = memory
tier1.channels.c1.capacity = 10000
tier1.channels.c1.transactionCapacity = 1000
tier1.sinks.sink1.channel      = c1
tier1.sinks.avrosink1.channel      = c1

tier1.channels.c2.type   = memory
tier1.channels.c2.capacity = 100000
tier1.channels.c2.transactionCapacity = 10000

#tier1.channels.c1.type   = file
#tier1.channels.c1.checkpointDir = /tmp/flume/checkpoint
#tier1.channels.c1.dataDirs = /tmp/flume/data
#tier1.sinks.avrosink1.channel      = c1
tier1.sinks.avrosink1.type = avro
tier1.sinks.avrosink1.hostname = ipaddr
tier1.sinks.avrosink1.port = 4444
tier1.sinks.avrosink1.avro.fileType = DataStream
tier1.sinks.avrosink1.serializer = avro_event
tier1.sinks.avrosink1.serializer.compressionCodec = snappy

tier1.sinks = filesink1
tier1.sinks.filesink1.type = file_roll
tier1.sinks.filesink1.channel = c2
tier1.sinks.filesink1.batchSize = 10000
#tier1.sinks.filesink1.rollInterval = 86400
tier1.sinks.filesink1.rollInterval =0
tier1.sinks.filesink1.sink.directory = /projects/logs

WARN http.HTTPSource: Error appending event to channel. Channel might be
full. Consider increasing the channel capacity or make sure the sinks
perform faster.
org.apache.flume.ChannelException: Unable to put batch on required channel:
org.apache.flume.channel.MemoryChannel{name: c1}
at
org.apache.flume.channel.ChannelProcessor.processEventBatch(ChannelProcessor.java:200)
at
org.apache.flume.source.http.HTTPSource$FlumeHTTPServlet.doPost(HTTPSource.java:201)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:725)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:814)
at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:401)
at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
at org.mortbay.jetty.Server.handle(Server.java:326)
at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
at
org.mortbay.jetty.HttpConnection$RequestHandler.content(HttpConnection.java:945)
at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:756)
at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:218)
at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
at
org.mortbay.jetty.bio.SocketConnector$Connection.run(SocketConnector.java:228)
at
org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
Caused by: org.apache.flume.ChannelException: Space for commit to queue
couldn't be acquired Sinks are likely not keeping up with sources, or the
buffer size is too tight
at
org.apache.flume.channel.MemoryChannel$MemoryTransaction.doCommit(MemoryChannel.java:128)
at
org.apache.flume.channel.BasicTransactionSemantics.commit(BasicTransactionSemantics.java:151)
at
org.apache.flume.channel.ChannelProcessor.processEventBatch(ChannelProcessor.java:192)
--
Deepak Subhramanian
+
Roshan Naik 2013-10-18, 22:55
+
Deepak Subhramanian 2013-10-23, 15:17