|
|
+
周梦想 2013-02-26, 05:57
+
周梦想 2013-02-26, 06:45
-
Re: Take list for MemoryTransaction, capacity 100 full?周梦想 2013-02-26, 07:41
I found if agent46.sinks.myhdfssink.hdfs.batchSize >= 100, it will report
this error. if I set this configure to 10, it's ok. but it's a bit slower. Best Regards, Andy 2013/2/26 周梦想 <[EMAIL PROTECTED]> > more logs: > > 2013-02-26 14:37:00,380 (SinkRunner-PollingRunner-DefaultSinkProcessor) > [ERROR - > org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:160)] Unable > to deliver event. Exception follows. > org.apache.flume.EventDeliveryException: > org.apache.flume.ChannelException: Take list for MemoryTransaction, > capacity 100 full, consider committing more frequently, increasing > capacity, or increasing thread count > at > org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:464) > at > org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68) > at > org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147) > at java.lang.Thread.run(Thread.java:722) > Caused by: org.apache.flume.ChannelException: Take list for > MemoryTransaction, capacity 100 full, consider committing more frequently, > increasing capacity, or increasing thread count > at > org.apache.flume.channel.MemoryChannel$MemoryTransaction.doTake(MemoryChannel.java:100) > at > org.apache.flume.channel.BasicTransactionSemantics.take(BasicTransactionSemantics.java:113) > at > org.apache.flume.channel.BasicChannelSemantics.take(BasicChannelSemantics.java:95) > at > org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:391) > ... 3 more > 2013-02-26 14:37:02,854 (pool-7-thread-1) [ERROR - > org.apache.flume.source.AvroSource.appendBatch(AvroSource.java:261)] Avro > source userlogsrc: Unable to process event batch. Exception follows. > org.apache.flume.ChannelException: Unable to put batch on required > channel: org.apache.flume.channel.MemoryChannel{name: memch1} > at > org.apache.flume.channel.ChannelProcessor.processEventBatch(ChannelProcessor.java:200) > at > org.apache.flume.source.AvroSource.appendBatch(AvroSource.java:259) > at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:601) > at > org.apache.avro.ipc.specific.SpecificResponder.respond(SpecificResponder.java:88) > at org.apache.avro.ipc.Responder.respond(Responder.java:149) > at > org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.messageReceived(NettyServer.java:188) > at > org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:75) > at > org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.handleUpstream(NettyServer.java:173) > at > org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) > at > org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:792) > at > org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296) > at > org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:321) > at > org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:303) > at > org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:208) > at > org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:75) > at > org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) > at > org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559) > at > org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268) > at > org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255) > at > org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:94) +
Hari Shreedharan 2013-02-26, 07:56
+
周梦想 2013-02-27, 02:29
|