Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Flume, mail # user - flume-ng Unable to put batch on required channel error


Copy link to this message
-
flume-ng Unable to put batch on required channel error
周梦想 2013-03-01, 02:16
hello,
I saw lot's of flume-ng 1.3.1 agent Exception said  Unable to put batch on
required channel,and then disconnected.
what is the effect? lost data or duplicate data?
What caused this error?
How to avoid this error?

some of my conf:
agent46.channels.memch1.type = memory
agent46.channels.memch1.capacity = 100000
agent46.channels.memch1.transactionCapactiy = 1000

agent46.sources.userlogsrc.type = avro

agent46.sinks.myhdfssink.type = hdfs
agent46.sinks.myhdfssink.hdfs.rollInterval = 620
agent46.sinks.myhdfssink.hdfs.appendTimeout = 1000
agent46.sinks.myhdfssink.hdfs.rollSize = 209715200
agent46.sinks.myhdfssink.hdfs.rollCount = 600000
agent46.sinks.myhdfssink.hdfs.batchSize = 50
agent46.sinks.myhdfssink.hdfs.txnEventMax = 100000
agent46.sinks.myhdfssink.hdfs.threadsPoolSize= 100

...
01 Mar 2013 03:00:16,368 ERROR [pool-7-thread-3]
(org.apache.flume.source.AvroSource.appendBatch:261)  - Avro source
gamelogsrc: Unable to process event batch. Exception follows.
org.apache.flume.ChannelException: Unable to put batch on required channel:
org.apache.flume.channel.MemoryChannel{name: memch1}
        at
org.apache.flume.channel.ChannelProcessor.processEventBatch(ChannelProcessor.java:200)
        at
org.apache.flume.source.AvroSource.appendBatch(AvroSource.java:259)
        at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source)
        at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:601)
        at
org.apache.avro.ipc.specific.SpecificResponder.respond(SpecificResponder.java:88)
        at org.apache.avro.ipc.Responder.respond(Responder.java:149)
        at
org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.messageReceived(NettyServer.java:188)
        at
org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:75)
        at
org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.handleUpstream(NettyServer.java:173)
        at
org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
        at
org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:792)
        at
org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296)
        at
org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:321)
        at
org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:303)
        at
org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:208)
        at
org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:75)
        at
org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
        at
org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
        at
org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268)
        at
org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255)
        at
org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:94)
        at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.processSelectedKeys(AbstractNioWorker.java:364)
        at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:238)
        at
org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
        at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
        at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
        at java.lang.Thread.run(Thread.java:722)
Caused by: org.apache.flume.ChannelException: Space for commit to queue
couldn't be acquired Sinks are likely not keeping up with sources, or the
buffer size is too tight
        at
org.apache.flume.channel.MemoryChannel$MemoryTransaction.doCommit(MemoryChannel.java:126)
        at
org.apache.flume.channel.BasicTransactionSemantics.commit(BasicTransactionSemantics.java:151)
        at
org.apache.flume.channel.ChannelProcessor.processEventBatch(ChannelProcessor.java:192)
        ... 28 more
01 Mar 2013 03:00:20,503 DEBUG [conf-file-poller-0]
(org.apache.flume.conf.file.AbstractFileConfigurationProvider$FileWatcherRunnable.run:188)
 - Checking file:conf/hdfs.conf for changes
01 Mar 2013 03:00:21,193 INFO  [pool-9-thread-1]
(org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.handleUpstream:171)
 - [id: 0x40044674, /192.168.133.44:2404 :> /192.168.133.47:5140]
DISCONNECTED
dleUpstream:171)  - [id: 0x40044674, /192.168.133.44:2404 :> /
192.168.133.47:5140] UNBOUND
01 Mar 2013 03:00:21,193 INFO  [pool-9-thread-1]
(org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.handleUpstream:171)
 - [id: 0x40044674, /192.168.133.44:2404 :> /192.168.133.47:5140] CLOSED
01 Mar 2013 03:00:21,194 INFO  [pool-9-thread-1]
(org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.channelClosed:209)
 - Connection to /192.168.133.44:2404 disconnected.

Thanks,
Andy