|
|
-
flume-ng Unable to put batch on required channel error
周梦想 2013-03-01, 02:16
hello,
I saw lot's of flume-ng 1.3.1 agent Exception said Unable to put batch on
required channel,and then disconnected.
what is the effect? lost data or duplicate data?
What caused this error?
How to avoid this error?
some of my conf:
agent46.channels.memch1.type = memory
agent46.channels.memch1.capacity = 100000
agent46.channels.memch1.transactionCapactiy = 1000
agent46.sources.userlogsrc.type = avro
agent46.sinks.myhdfssink.type = hdfs
agent46.sinks.myhdfssink.hdfs.rollInterval = 620
agent46.sinks.myhdfssink.hdfs.appendTimeout = 1000
agent46.sinks.myhdfssink.hdfs.rollSize = 209715200
agent46.sinks.myhdfssink.hdfs.rollCount = 600000
agent46.sinks.myhdfssink.hdfs.batchSize = 50
agent46.sinks.myhdfssink.hdfs.txnEventMax = 100000
agent46.sinks.myhdfssink.hdfs.threadsPoolSize= 100
...
01 Mar 2013 03:00:16,368 ERROR [pool-7-thread-3]
(org.apache.flume.source.AvroSource.appendBatch:261) - Avro source
gamelogsrc: Unable to process event batch. Exception follows.
org.apache.flume.ChannelException: Unable to put batch on required channel:
org.apache.flume.channel.MemoryChannel{name: memch1}
at
org.apache.flume.channel.ChannelProcessor.processEventBatch(ChannelProcessor.java:200)
at
org.apache.flume.source.AvroSource.appendBatch(AvroSource.java:259)
at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at
org.apache.avro.ipc.specific.SpecificResponder.respond(SpecificResponder.java:88)
at org.apache.avro.ipc.Responder.respond(Responder.java:149)
at
org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.messageReceived(NettyServer.java:188)
at
org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:75)
at
org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.handleUpstream(NettyServer.java:173)
at
org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
at
org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:792)
at
org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296)
at
org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:321)
at
org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:303)
at
org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:208)
at
org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:75)
at
org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
at
org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
at
org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268)
at
org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255)
at
org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:94)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.processSelectedKeys(AbstractNioWorker.java:364)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:238)
at
org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:722)
Caused by: org.apache.flume.ChannelException: Space for commit to queue
couldn't be acquired Sinks are likely not keeping up with sources, or the
buffer size is too tight
at
org.apache.flume.channel.MemoryChannel$MemoryTransaction.doCommit(MemoryChannel.java:126)
at
org.apache.flume.channel.BasicTransactionSemantics.commit(BasicTransactionSemantics.java:151)
at
org.apache.flume.channel.ChannelProcessor.processEventBatch(ChannelProcessor.java:192)
... 28 more
01 Mar 2013 03:00:20,503 DEBUG [conf-file-poller-0]
(org.apache.flume.conf.file.AbstractFileConfigurationProvider$FileWatcherRunnable.run:188)
- Checking file:conf/hdfs.conf for changes
01 Mar 2013 03:00:21,193 INFO [pool-9-thread-1]
(org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.handleUpstream:171)
- [id: 0x40044674, /192.168.133.44:2404 :> /192.168.133.47:5140]
DISCONNECTED
dleUpstream:171) - [id: 0x40044674, /192.168.133.44:2404 :> /
192.168.133.47:5140] UNBOUND
01 Mar 2013 03:00:21,193 INFO [pool-9-thread-1]
(org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.handleUpstream:171)
- [id: 0x40044674, /192.168.133.44:2404 :> /192.168.133.47:5140] CLOSED
01 Mar 2013 03:00:21,194 INFO [pool-9-thread-1]
(org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.channelClosed:209)
- Connection to /192.168.133.44:2404 disconnected.
Thanks,
Andy
-
Re: flume-ng Unable to put batch on required channel error
周梦想 2013-03-01, 03:10
Hello,
I wrote a test java file to put local files to hdfs, it put 1.1GB files to
hdfs spent about 35 seconds. while using flume, 100MB files it should spend
about 3-5 minutes.
why flume is so slow and always report error? how to enlarge the flume hdfs
sink throughput?
Andy
2013/3/1 周梦想 <[EMAIL PROTECTED]>
> hello,
> I saw lot's of flume-ng 1.3.1 agent Exception said Unable to put batch on
> required channel,and then disconnected.
> what is the effect? lost data or duplicate data?
> What caused this error?
> How to avoid this error?
>
> some of my conf:
> agent46.channels.memch1.type = memory
> agent46.channels.memch1.capacity = 100000
> agent46.channels.memch1.transactionCapactiy = 1000
>
> agent46.sources.userlogsrc.type = avro
>
> agent46.sinks.myhdfssink.type = hdfs
> agent46.sinks.myhdfssink.hdfs.rollInterval = 620
> agent46.sinks.myhdfssink.hdfs.appendTimeout = 1000
> agent46.sinks.myhdfssink.hdfs.rollSize = 209715200
> agent46.sinks.myhdfssink.hdfs.rollCount = 600000
> agent46.sinks.myhdfssink.hdfs.batchSize = 50
> agent46.sinks.myhdfssink.hdfs.txnEventMax = 100000
> agent46.sinks.myhdfssink.hdfs.threadsPoolSize= 100
>
> ...
> 01 Mar 2013 03:00:16,368 ERROR [pool-7-thread-3]
> (org.apache.flume.source.AvroSource.appendBatch:261) - Avro source
> gamelogsrc: Unable to process event batch. Exception follows.
> org.apache.flume.ChannelException: Unable to put batch on required
> channel: org.apache.flume.channel.MemoryChannel{name: memch1}
> at
> org.apache.flume.channel.ChannelProcessor.processEventBatch(ChannelProcessor.java:200)
> at
> org.apache.flume.source.AvroSource.appendBatch(AvroSource.java:259)
> at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:601)
> at
> org.apache.avro.ipc.specific.SpecificResponder.respond(SpecificResponder.java:88)
> at org.apache.avro.ipc.Responder.respond(Responder.java:149)
> at
> org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.messageReceived(NettyServer.java:188)
> at
> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:75)
> at
> org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.handleUpstream(NettyServer.java:173)
> at
> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
> at
> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:792)
> at
> org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296)
> at
> org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:321)
> at
> org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:303)
> at
> org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:208)
> at
> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:75)
> at
> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
> at
> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
> at
> org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268)
> at
> org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255)
> at
> org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:94)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.processSelectedKeys(AbstractNioWorker.java:364)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:238)
> at
> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
-
Re: flume-ng Unable to put batch on required channel error
Hari Shreedharan 2013-03-01, 05:24
Try increasing the callTimeout for HDFS Sink - that is in milliseconds. Also, you should try using a batch size/txnEvent max of at least 100 (or more). Using more HDFS sinks will also improve performance by increased parallel writes since each sink is single threaded and writes to only one file at a time. (note that each HDFS Sink should write to a different directory).
--
Hari Shreedharan
On Thursday, February 28, 2013 at 7:10 PM, 周梦想 wrote:
> Hello,
>
> I wrote a test java file to put local files to hdfs, it put 1.1GB files to hdfs spent about 35 seconds. while using flume, 100MB files it should spend about 3-5 minutes.
> why flume is so slow and always report error? how to enlarge the flume hdfs sink throughput?
>
> Andy
>
> 2013/3/1 周梦想 <[EMAIL PROTECTED] (mailto:[EMAIL PROTECTED])>
> > hello,
> > I saw lot's of flume-ng 1.3.1 agent Exception said Unable to put batch on required channel,and then disconnected.
> > what is the effect? lost data or duplicate data?
> > What caused this error?
> > How to avoid this error?
> >
> > some of my conf:
> > agent46.channels.memch1.type = memory
> > agent46.channels.memch1.capacity = 100000
> > agent46.channels.memch1.transactionCapactiy = 1000
> >
> >
> > agent46.sources.userlogsrc.type = avro
> >
> > agent46.sinks.myhdfssink.type = hdfs
> > agent46.sinks.myhdfssink.hdfs.rollInterval = 620
> > agent46.sinks.myhdfssink.hdfs.appendTimeout = 1000
> > agent46.sinks.myhdfssink.hdfs.rollSize = 209715200
> > agent46.sinks.myhdfssink.hdfs.rollCount = 600000
> > agent46.sinks.myhdfssink.hdfs.batchSize = 50
> > agent46.sinks.myhdfssink.hdfs.txnEventMax = 100000
> > agent46.sinks.myhdfssink.hdfs.threadsPoolSize= 100
> >
> >
> > ...
> > 01 Mar 2013 03:00:16,368 ERROR [pool-7-thread-3] (org.apache.flume.source.AvroSource.appendBatch:261) - Avro source gamelogsrc: Unable to process event batch. Exception follows.
> > org.apache.flume.ChannelException: Unable to put batch on required channel: org.apache.flume.channel.MemoryChannel{name: memch1}
> > at org.apache.flume.channel.ChannelProcessor.processEventBatch(ChannelProcessor.java:200)
> > at org.apache.flume.source.AvroSource.appendBatch(AvroSource.java:259)
> > at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source)
> > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> > at java.lang.reflect.Method.invoke(Method.java:601)
> > at org.apache.avro.ipc.specific.SpecificResponder.respond(SpecificResponder.java:88)
> > at org.apache.avro.ipc.Responder.respond(Responder.java:149)
> > at org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.messageReceived(NettyServer.java:188)
> > at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:75)
> > at org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.handleUpstream(NettyServer.java:173)
> > at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
> > at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:792)
> > at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296)
> > at org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:321)
> > at org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:303)
> > at org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:208)
> > at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:75)
> > at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
> > at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
> > at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268)
+
Hari Shreedharan 2013-03-01, 05:24
-
Re: flume-ng Unable to put batch on required channel error
周梦想 2013-03-01, 09:54
thank you. I once increased hdfs batch size more than 1000, but it report
the ChannelException more frequence. so I changed small and small, batch
size to 50 is less error.
2013/3/1 Hari Shreedharan <[EMAIL PROTECTED]>
> Try increasing the callTimeout for HDFS Sink - that is in milliseconds.
> Also, you should try using a batch size/txnEvent max of at least 100 (or
> more). Using more HDFS sinks will also improve performance by increased
> parallel writes since each sink is single threaded and writes to only one
> file at a time. (note that each HDFS Sink should write to a different
> directory).
>
> --
> Hari Shreedharan
>
> On Thursday, February 28, 2013 at 7:10 PM, 周梦想 wrote:
>
> Hello,
>
> I wrote a test java file to put local files to hdfs, it put 1.1GB files to
> hdfs spent about 35 seconds. while using flume, 100MB files it should spend
> about 3-5 minutes.
> why flume is so slow and always report error? how to enlarge the flume
> hdfs sink throughput?
>
> Andy
>
> 2013/3/1 周梦想 <[EMAIL PROTECTED]>
>
> hello,
> I saw lot's of flume-ng 1.3.1 agent Exception said Unable to put batch on
> required channel,and then disconnected.
> what is the effect? lost data or duplicate data?
> What caused this error?
> How to avoid this error?
>
> some of my conf:
> agent46.channels.memch1.type = memory
> agent46.channels.memch1.capacity = 100000
> agent46.channels.memch1.transactionCapactiy = 1000
>
> agent46.sources.userlogsrc.type = avro
>
> agent46.sinks.myhdfssink.type = hdfs
> agent46.sinks.myhdfssink.hdfs.rollInterval = 620
> agent46.sinks.myhdfssink.hdfs.appendTimeout = 1000
> agent46.sinks.myhdfssink.hdfs.rollSize = 209715200
> agent46.sinks.myhdfssink.hdfs.rollCount = 600000
> agent46.sinks.myhdfssink.hdfs.batchSize = 50
> agent46.sinks.myhdfssink.hdfs.txnEventMax = 100000
> agent46.sinks.myhdfssink.hdfs.threadsPoolSize= 100
>
> ...
> 01 Mar 2013 03:00:16,368 ERROR [pool-7-thread-3]
> (org.apache.flume.source.AvroSource.appendBatch:261) - Avro source
> gamelogsrc: Unable to process event batch. Exception follows.
> org.apache.flume.ChannelException: Unable to put batch on required
> channel: org.apache.flume.channel.MemoryChannel{name: memch1}
> at
> org.apache.flume.channel.ChannelProcessor.processEventBatch(ChannelProcessor.java:200)
> at
> org.apache.flume.source.AvroSource.appendBatch(AvroSource.java:259)
> at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:601)
> at
> org.apache.avro.ipc.specific.SpecificResponder.respond(SpecificResponder.java:88)
> at org.apache.avro.ipc.Responder.respond(Responder.java:149)
> at
> org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.messageReceived(NettyServer.java:188)
> at
> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:75)
> at
> org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.handleUpstream(NettyServer.java:173)
> at
> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
> at
> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:792)
> at
> org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296)
> at
> org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:321)
> at
> org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:303)
> at
> org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:208)
> at
> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:75)
> at
> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
> at
|
|
