Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Flume >> mail # user >> Take list for MemoryTransaction, capacity 100 full ?


+
周梦想 2013-02-26, 05:57
+
周梦想 2013-02-26, 06:45
+
周梦想 2013-02-26, 07:41
Copy link to this message
-
Re: Take list for MemoryTransaction, capacity 100 full?
This is because the memory channel has a default transaction capacity of 100. Increasing it (or keeping sinks's batchSize < transaction capacity of the channel will fix the issue). See http://flume.apache.org/FlumeUserGuide.html#memory-channel for more details.

Hari  

--  
Hari Shreedharan
On Monday, February 25, 2013 at 11:41 PM, 周梦想 wrote:

> I found if agent46.sinks.myhdfssink.hdfs.batchSize >= 100, it will report this error.
> if I set this configure to 10, it's ok. but it's a bit slower.
>  
> Best Regards,
> Andy
>  
> 2013/2/26 周梦想 <[EMAIL PROTECTED] (mailto:[EMAIL PROTECTED])>
> > more logs:
> >  
> > 2013-02-26 14:37:00,380 (SinkRunner-PollingRunner-DefaultSinkProcessor) [ERROR - org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:160)] Unable to deliver event. Exception follows.  
> > org.apache.flume.EventDeliveryException: org.apache.flume.ChannelException: Take list for MemoryTransaction, capacity 100 full, consider committing more frequently, increasing capacity, or increasing thread count
> >         at org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:464)
> >         at org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68)
> >         at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
> >         at java.lang.Thread.run(Thread.java:722)
> >  
> > Caused by: org.apache.flume.ChannelException: Take list for MemoryTransaction, capacity 100 full, consider committing more frequently, increasing capacity, or increasing thread count
> >         at org.apache.flume.channel.MemoryChannel$MemoryTransaction.doTake(MemoryChannel.java:100)
> >         at org.apache.flume.channel.BasicTransactionSemantics.take(BasicTransactionSemantics.java:113)
> >         at org.apache.flume.channel.BasicChannelSemantics.take(BasicChannelSemantics.java:95)
> >         at org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:391)
> >  
> >         ... 3 more
> > 2013-02-26 14:37:02,854 (pool-7-thread-1) [ERROR - org.apache.flume.source.AvroSource.appendBatch(AvroSource.java:261)] Avro source userlogsrc: Unable to process event batch. Exception follows.
> > org.apache.flume.ChannelException: Unable to put batch on required channel: org.apache.flume.channel.MemoryChannel{name: memch1}
> >         at org.apache.flume.channel.ChannelProcessor.processEventBatch(ChannelProcessor.java:200)
> >         at org.apache.flume.source.AvroSource.appendBatch(AvroSource.java:259)
> >         at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source)
> >         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> >         at java.lang.reflect.Method.invoke(Method.java:601)
> >         at org.apache.avro.ipc.specific.SpecificResponder.respond(SpecificResponder.java:88)
> >         at org.apache.avro.ipc.Responder.respond(Responder.java:149)
> >         at org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.messageReceived(NettyServer.java:188)
> >         at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:75)
> >         at org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.handleUpstream(NettyServer.java:173)
> >         at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
> >         at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:792)
> >         at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296)
> >         at org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:321)
> >         at org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:303)
> >         at org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:208)
> >         at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:75)
+
周梦想 2013-02-27, 02:29