Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Flume >> mail # user >> Take list for MemoryTransaction, capacity 100 full ?


+
周梦想 2013-02-26, 05:57
+
周梦想 2013-02-26, 06:45
+
周梦想 2013-02-26, 07:41
Copy link to this message
-
Re: Take list for MemoryTransaction, capacity 100 full?
This is because the memory channel has a default transaction capacity of 100. Increasing it (or keeping sinks's batchSize < transaction capacity of the channel will fix the issue). See http://flume.apache.org/FlumeUserGuide.html#memory-channel for more details.

Hari  

--  
Hari Shreedharan
On Monday, February 25, 2013 at 11:41 PM, 周梦想 wrote:

> I found if agent46.sinks.myhdfssink.hdfs.batchSize >= 100, it will report this error.
> if I set this configure to 10, it's ok. but it's a bit slower.
>  
> Best Regards,
> Andy
>  
> 2013/2/26 周梦想 <[EMAIL PROTECTED] (mailto:[EMAIL PROTECTED])>
> > more logs:
> >  
> > 2013-02-26 14:37:00,380 (SinkRunner-PollingRunner-DefaultSinkProcessor) [ERROR - org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:160)] Unable to deliver event. Exception follows.  
> > org.apache.flume.EventDeliveryException: org.apache.flume.ChannelException: Take list for MemoryTransaction, capacity 100 full, consider committing more frequently, increasing capacity, or increasing thread count
> >         at org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:464)
> >         at org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68)
> >         at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
> >         at java.lang.Thread.run(Thread.java:722)
> >  
> > Caused by: org.apache.flume.ChannelException: Take list for MemoryTransaction, capacity 100 full, consider committing more frequently, increasing capacity, or increasing thread count
> >         at org.apache.flume.channel.MemoryChannel$MemoryTransaction.doTake(MemoryChannel.java:100)
> >         at org.apache.flume.channel.BasicTransactionSemantics.take(BasicTransactionSemantics.java:113)
> >         at org.apache.flume.channel.BasicChannelSemantics.take(BasicChannelSemantics.java:95)
> >         at org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:391)
> >  
> >         ... 3 more
> > 2013-02-26 14:37:02,854 (pool-7-thread-1) [ERROR - org.apache.flume.source.AvroSource.appendBatch(AvroSource.java:261)] Avro source userlogsrc: Unable to process event batch. Exception follows.
> > org.apache.flume.ChannelException: Unable to put batch on required channel: org.apache.flume.channel.MemoryChannel{name: memch1}
> >         at org.apache.flume.channel.ChannelProcessor.processEventBatch(ChannelProcessor.java:200)
> >         at org.apache.flume.source.AvroSource.appendBatch(AvroSource.java:259)
> >         at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source)
> >         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> >         at java.lang.reflect.Method.invoke(Method.java:601)
> >         at org.apache.avro.ipc.specific.SpecificResponder.respond(SpecificResponder.java:88)
> >         at org.apache.avro.ipc.Responder.respond(Responder.java:149)
> >         at org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.messageReceived(NettyServer.java:188)
> >         at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:75)
> >         at org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.handleUpstream(NettyServer.java:173)
> >         at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
> >         at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:792)
> >         at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296)
> >         at org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:321)
> >         at org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:303)
> >         at org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:208)
> >         at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:75)
+
周梦想 2013-02-27, 02:29
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB