Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Flume >> mail # user >> Memory Channel Error on Load Testing


+
Deepak Subhramanian 2013-10-18, 15:47
+
Roshan Naik 2013-10-18, 22:55
Copy link to this message
-
Re: Memory Channel Error on Load Testing
Thanks Roshan. I increased the heapsize and it worked fine.
On Fri, Oct 18, 2013 at 11:55 PM, Roshan Naik <[EMAIL PROTECTED]>wrote:

> the error indicates that source is pumping in data faster than the sink is
> draining the memory channel. causing the channel to fill up. it does not
> appear to be a MemCh issue.
>
>
> On Fri, Oct 18, 2013 at 8:47 AM, Deepak Subhramanian <
> [EMAIL PROTECTED]> wrote:
>
>> Hi
>> I am getting error while load testing a flume agent with HTTSrc and Avro
>> Sink.
>>
>> Is there an optimum configuration for memory channel for get flume
>> working on high load. Thanks in advance for any inputs.
>>
>>
>> tier1.sources  = httpsrc1
>> tier1.channels = c1   c2
>> tier1.sinks    = avrosink1
>>
>> tier1.sources.httpsrc1.bind     = host1
>> tier1.sources.httpsrc1.type = http
>> tier1.sources.httpsrc1.port = 443
>> tier1.sources.httpsrc1.channels = c1 c2
>> tier1.sources.httpsrc1.handler =flume.SimpleHTTPEventHandler
>> tier1.sources.httpsrc1.handler.nickname = HTTPTesting
>>
>> tier1.channels.c1.type   = memory
>> tier1.channels.c1.capacity = 10000
>> tier1.channels.c1.transactionCapacity = 1000
>> tier1.sinks.sink1.channel      = c1
>> tier1.sinks.avrosink1.channel      = c1
>>
>> tier1.channels.c2.type   = memory
>> tier1.channels.c2.capacity = 100000
>> tier1.channels.c2.transactionCapacity = 10000
>>
>> #tier1.channels.c1.type   = file
>>  #tier1.channels.c1.checkpointDir = /tmp/flume/checkpoint
>> #tier1.channels.c1.dataDirs = /tmp/flume/data
>> #tier1.sinks.avrosink1.channel      = c1
>>
>>
>> tier1.sinks.avrosink1.type = avro
>> tier1.sinks.avrosink1.hostname = ipaddr
>> tier1.sinks.avrosink1.port = 4444
>> tier1.sinks.avrosink1.avro.fileType = DataStream
>> tier1.sinks.avrosink1.serializer = avro_event
>> tier1.sinks.avrosink1.serializer.compressionCodec = snappy
>>
>>
>>
>> tier1.sinks = filesink1
>> tier1.sinks.filesink1.type = file_roll
>> tier1.sinks.filesink1.channel = c2
>> tier1.sinks.filesink1.batchSize = 10000
>>  #tier1.sinks.filesink1.rollInterval = 86400
>> tier1.sinks.filesink1.rollInterval =0
>> tier1.sinks.filesink1.sink.directory = /projects/logs
>>
>>
>>
>> WARN http.HTTPSource: Error appending event to channel. Channel might be
>> full. Consider increasing the channel capacity or make sure the sinks
>> perform faster.
>> org.apache.flume.ChannelException: Unable to put batch on required
>> channel: org.apache.flume.channel.MemoryChannel{name: c1}
>>  at
>> org.apache.flume.channel.ChannelProcessor.processEventBatch(ChannelProcessor.java:200)
>> at
>> org.apache.flume.source.http.HTTPSource$FlumeHTTPServlet.doPost(HTTPSource.java:201)
>>  at javax.servlet.http.HttpServlet.service(HttpServlet.java:725)
>> at javax.servlet.http.HttpServlet.service(HttpServlet.java:814)
>>  at
>> org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
>> at
>> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:401)
>>  at
>> org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
>> at
>> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
>>  at
>> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
>> at org.mortbay.jetty.Server.handle(Server.java:326)
>>  at
>> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
>> at
>> org.mortbay.jetty.HttpConnection$RequestHandler.content(HttpConnection.java:945)
>>  at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:756)
>> at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:218)
>>  at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
>> at
>> org.mortbay.jetty.bio.SocketConnector$Connection.run(SocketConnector.java:228)
>>  at
>> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
>> Caused by: org.apache.flume.ChannelException: Space for commit to queue
>> couldn't be acquired Sinks are likely not keeping up with sources, or the
>> buffer size is too tight
>>  at
>> org.apache.flume.channel.MemoryChannel$MemoryTransaction.doCommit(MemoryChannel.java:128)
Deepak Subhramanian