Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Flume >> mail # user >> Memory Channel Error on Load Testing


Copy link to this message
-
Re: Memory Channel Error on Load Testing
the error indicates that source is pumping in data faster than the sink is
draining the memory channel. causing the channel to fill up. it does not
appear to be a MemCh issue.
On Fri, Oct 18, 2013 at 8:47 AM, Deepak Subhramanian <
[EMAIL PROTECTED]> wrote:

> Hi
> I am getting error while load testing a flume agent with HTTSrc and Avro
> Sink.
>
> Is there an optimum configuration for memory channel for get flume working
> on high load. Thanks in advance for any inputs.
>
>
> tier1.sources  = httpsrc1
> tier1.channels = c1   c2
> tier1.sinks    = avrosink1
>
> tier1.sources.httpsrc1.bind     = host1
> tier1.sources.httpsrc1.type = http
> tier1.sources.httpsrc1.port = 443
> tier1.sources.httpsrc1.channels = c1 c2
> tier1.sources.httpsrc1.handler =flume.SimpleHTTPEventHandler
> tier1.sources.httpsrc1.handler.nickname = HTTPTesting
>
> tier1.channels.c1.type   = memory
> tier1.channels.c1.capacity = 10000
> tier1.channels.c1.transactionCapacity = 1000
> tier1.sinks.sink1.channel      = c1
> tier1.sinks.avrosink1.channel      = c1
>
> tier1.channels.c2.type   = memory
> tier1.channels.c2.capacity = 100000
> tier1.channels.c2.transactionCapacity = 10000
>
> #tier1.channels.c1.type   = file
> #tier1.channels.c1.checkpointDir = /tmp/flume/checkpoint
> #tier1.channels.c1.dataDirs = /tmp/flume/data
> #tier1.sinks.avrosink1.channel      = c1
>
>
> tier1.sinks.avrosink1.type = avro
> tier1.sinks.avrosink1.hostname = ipaddr
> tier1.sinks.avrosink1.port = 4444
> tier1.sinks.avrosink1.avro.fileType = DataStream
> tier1.sinks.avrosink1.serializer = avro_event
> tier1.sinks.avrosink1.serializer.compressionCodec = snappy
>
>
>
> tier1.sinks = filesink1
> tier1.sinks.filesink1.type = file_roll
> tier1.sinks.filesink1.channel = c2
> tier1.sinks.filesink1.batchSize = 10000
> #tier1.sinks.filesink1.rollInterval = 86400
> tier1.sinks.filesink1.rollInterval =0
> tier1.sinks.filesink1.sink.directory = /projects/logs
>
>
>
> WARN http.HTTPSource: Error appending event to channel. Channel might be
> full. Consider increasing the channel capacity or make sure the sinks
> perform faster.
> org.apache.flume.ChannelException: Unable to put batch on required
> channel: org.apache.flume.channel.MemoryChannel{name: c1}
>  at
> org.apache.flume.channel.ChannelProcessor.processEventBatch(ChannelProcessor.java:200)
> at
> org.apache.flume.source.http.HTTPSource$FlumeHTTPServlet.doPost(HTTPSource.java:201)
>  at javax.servlet.http.HttpServlet.service(HttpServlet.java:725)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:814)
>  at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
> at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:401)
>  at
> org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
> at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
>  at
> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
> at org.mortbay.jetty.Server.handle(Server.java:326)
>  at
> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
> at
> org.mortbay.jetty.HttpConnection$RequestHandler.content(HttpConnection.java:945)
>  at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:756)
> at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:218)
>  at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
> at
> org.mortbay.jetty.bio.SocketConnector$Connection.run(SocketConnector.java:228)
>  at
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
> Caused by: org.apache.flume.ChannelException: Space for commit to queue
> couldn't be acquired Sinks are likely not keeping up with sources, or the
> buffer size is too tight
>  at
> org.apache.flume.channel.MemoryChannel$MemoryTransaction.doCommit(MemoryChannel.java:128)
> at
> org.apache.flume.channel.BasicTransactionSemantics.commit(BasicTransactionSemantics.java:151)
>  at
> org.apache.flume.channel.ChannelProcessor.processEventBatch(ChannelProcessor.java:192)

CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.