Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Flume >> mail # user >> Take list for MemoryTransaction, capacity 100 full ?


+
周梦想 2013-02-26, 05:57
+
周梦想 2013-02-26, 06:45
Copy link to this message
-
Re: Take list for MemoryTransaction, capacity 100 full?
I found if agent46.sinks.myhdfssink.hdfs.batchSize >= 100, it will report
this error.
if I set this configure to 10, it's ok. but it's a bit slower.

Best Regards,
Andy

2013/2/26 周梦想 <[EMAIL PROTECTED]>

> more logs:
>
> 2013-02-26 14:37:00,380 (SinkRunner-PollingRunner-DefaultSinkProcessor)
> [ERROR -
> org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:160)] Unable
> to deliver event. Exception follows.
> org.apache.flume.EventDeliveryException:
> org.apache.flume.ChannelException: Take list for MemoryTransaction,
> capacity 100 full, consider committing more frequently, increasing
> capacity, or increasing thread count
>         at
> org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:464)
>         at
> org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68)
>         at
> org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
>         at java.lang.Thread.run(Thread.java:722)
> Caused by: org.apache.flume.ChannelException: Take list for
> MemoryTransaction, capacity 100 full, consider committing more frequently,
> increasing capacity, or increasing thread count
>          at
> org.apache.flume.channel.MemoryChannel$MemoryTransaction.doTake(MemoryChannel.java:100)
>         at
> org.apache.flume.channel.BasicTransactionSemantics.take(BasicTransactionSemantics.java:113)
>         at
> org.apache.flume.channel.BasicChannelSemantics.take(BasicChannelSemantics.java:95)
>         at
> org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:391)
>         ... 3 more
> 2013-02-26 14:37:02,854 (pool-7-thread-1) [ERROR -
> org.apache.flume.source.AvroSource.appendBatch(AvroSource.java:261)] Avro
> source userlogsrc: Unable to process event batch. Exception follows.
> org.apache.flume.ChannelException: Unable to put batch on required
> channel: org.apache.flume.channel.MemoryChannel{name: memch1}
>         at
> org.apache.flume.channel.ChannelProcessor.processEventBatch(ChannelProcessor.java:200)
>         at
> org.apache.flume.source.AvroSource.appendBatch(AvroSource.java:259)
>         at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source)
>         at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>         at java.lang.reflect.Method.invoke(Method.java:601)
>         at
> org.apache.avro.ipc.specific.SpecificResponder.respond(SpecificResponder.java:88)
>         at org.apache.avro.ipc.Responder.respond(Responder.java:149)
>         at
> org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.messageReceived(NettyServer.java:188)
>         at
> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:75)
>         at
> org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.handleUpstream(NettyServer.java:173)
>         at
> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
>         at
> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:792)
>         at
> org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296)
>         at
> org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:321)
>         at
> org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:303)
>         at
> org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:208)
>         at
> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:75)
>         at
> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
>         at
> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
>         at
> org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268)
>         at
> org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255)
>         at
> org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:94)
+
Hari Shreedharan 2013-02-26, 07:56
+
周梦想 2013-02-27, 02:29
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB