Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Flume >> mail # user >> Memory Channel Error on Load Testing


+
Deepak Subhramanian 2013-10-18, 15:47
+
Roshan Naik 2013-10-18, 22:55
Copy link to this message
-
Re: Memory Channel Error on Load Testing
Thanks Roshan. I increased the heapsize and it worked fine.
On Fri, Oct 18, 2013 at 11:55 PM, Roshan Naik <[EMAIL PROTECTED]>wrote:

> the error indicates that source is pumping in data faster than the sink is
> draining the memory channel. causing the channel to fill up. it does not
> appear to be a MemCh issue.
>
>
> On Fri, Oct 18, 2013 at 8:47 AM, Deepak Subhramanian <
> [EMAIL PROTECTED]> wrote:
>
>> Hi
>> I am getting error while load testing a flume agent with HTTSrc and Avro
>> Sink.
>>
>> Is there an optimum configuration for memory channel for get flume
>> working on high load. Thanks in advance for any inputs.
>>
>>
>> tier1.sources  = httpsrc1
>> tier1.channels = c1   c2
>> tier1.sinks    = avrosink1
>>
>> tier1.sources.httpsrc1.bind     = host1
>> tier1.sources.httpsrc1.type = http
>> tier1.sources.httpsrc1.port = 443
>> tier1.sources.httpsrc1.channels = c1 c2
>> tier1.sources.httpsrc1.handler =flume.SimpleHTTPEventHandler
>> tier1.sources.httpsrc1.handler.nickname = HTTPTesting
>>
>> tier1.channels.c1.type   = memory
>> tier1.channels.c1.capacity = 10000
>> tier1.channels.c1.transactionCapacity = 1000
>> tier1.sinks.sink1.channel      = c1
>> tier1.sinks.avrosink1.channel      = c1
>>
>> tier1.channels.c2.type   = memory
>> tier1.channels.c2.capacity = 100000
>> tier1.channels.c2.transactionCapacity = 10000
>>
>> #tier1.channels.c1.type   = file
>>  #tier1.channels.c1.checkpointDir = /tmp/flume/checkpoint
>> #tier1.channels.c1.dataDirs = /tmp/flume/data
>> #tier1.sinks.avrosink1.channel      = c1
>>
>>
>> tier1.sinks.avrosink1.type = avro
>> tier1.sinks.avrosink1.hostname = ipaddr
>> tier1.sinks.avrosink1.port = 4444
>> tier1.sinks.avrosink1.avro.fileType = DataStream
>> tier1.sinks.avrosink1.serializer = avro_event
>> tier1.sinks.avrosink1.serializer.compressionCodec = snappy
>>
>>
>>
>> tier1.sinks = filesink1
>> tier1.sinks.filesink1.type = file_roll
>> tier1.sinks.filesink1.channel = c2
>> tier1.sinks.filesink1.batchSize = 10000
>>  #tier1.sinks.filesink1.rollInterval = 86400
>> tier1.sinks.filesink1.rollInterval =0
>> tier1.sinks.filesink1.sink.directory = /projects/logs
>>
>>
>>
>> WARN http.HTTPSource: Error appending event to channel. Channel might be
>> full. Consider increasing the channel capacity or make sure the sinks
>> perform faster.
>> org.apache.flume.ChannelException: Unable to put batch on required
>> channel: org.apache.flume.channel.MemoryChannel{name: c1}
>>  at
>> org.apache.flume.channel.ChannelProcessor.processEventBatch(ChannelProcessor.java:200)
>> at
>> org.apache.flume.source.http.HTTPSource$FlumeHTTPServlet.doPost(HTTPSource.java:201)
>>  at javax.servlet.http.HttpServlet.service(HttpServlet.java:725)
>> at javax.servlet.http.HttpServlet.service(HttpServlet.java:814)
>>  at
>> org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
>> at
>> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:401)
>>  at
>> org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
>> at
>> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
>>  at
>> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
>> at org.mortbay.jetty.Server.handle(Server.java:326)
>>  at
>> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
>> at
>> org.mortbay.jetty.HttpConnection$RequestHandler.content(HttpConnection.java:945)
>>  at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:756)
>> at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:218)
>>  at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
>> at
>> org.mortbay.jetty.bio.SocketConnector$Connection.run(SocketConnector.java:228)
>>  at
>> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
>> Caused by: org.apache.flume.ChannelException: Space for commit to queue
>> couldn't be acquired Sinks are likely not keeping up with sources, or the
>> buffer size is too tight
>>  at
>> org.apache.flume.channel.MemoryChannel$MemoryTransaction.doCommit(MemoryChannel.java:128)
Deepak Subhramanian
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB