Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Flume >> mail # user >> flume-ng Unable to put batch on required channel error


Copy link to this message
-
Re: flume-ng Unable to put batch on required channel error
Hello,

I wrote a test java file to put local files to hdfs, it put 1.1GB files to
hdfs spent about 35 seconds. while using flume, 100MB files it should spend
about 3-5 minutes.
why flume is so slow and always report error? how to enlarge the flume hdfs
sink throughput?

Andy

2013/3/1 周梦想 <[EMAIL PROTECTED]>

> hello,
> I saw lot's of flume-ng 1.3.1 agent Exception said  Unable to put batch on
> required channel,and then disconnected.
> what is the effect? lost data or duplicate data?
> What caused this error?
> How to avoid this error?
>
> some of my conf:
> agent46.channels.memch1.type = memory
> agent46.channels.memch1.capacity = 100000
> agent46.channels.memch1.transactionCapactiy = 1000
>
> agent46.sources.userlogsrc.type = avro
>
> agent46.sinks.myhdfssink.type = hdfs
> agent46.sinks.myhdfssink.hdfs.rollInterval = 620
> agent46.sinks.myhdfssink.hdfs.appendTimeout = 1000
> agent46.sinks.myhdfssink.hdfs.rollSize = 209715200
> agent46.sinks.myhdfssink.hdfs.rollCount = 600000
> agent46.sinks.myhdfssink.hdfs.batchSize = 50
> agent46.sinks.myhdfssink.hdfs.txnEventMax = 100000
> agent46.sinks.myhdfssink.hdfs.threadsPoolSize= 100
>
> ...
> 01 Mar 2013 03:00:16,368 ERROR [pool-7-thread-3]
> (org.apache.flume.source.AvroSource.appendBatch:261)  - Avro source
> gamelogsrc: Unable to process event batch. Exception follows.
> org.apache.flume.ChannelException: Unable to put batch on required
> channel: org.apache.flume.channel.MemoryChannel{name: memch1}
>         at
> org.apache.flume.channel.ChannelProcessor.processEventBatch(ChannelProcessor.java:200)
>         at
> org.apache.flume.source.AvroSource.appendBatch(AvroSource.java:259)
>         at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source)
>         at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>         at java.lang.reflect.Method.invoke(Method.java:601)
>         at
> org.apache.avro.ipc.specific.SpecificResponder.respond(SpecificResponder.java:88)
>         at org.apache.avro.ipc.Responder.respond(Responder.java:149)
>         at
> org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.messageReceived(NettyServer.java:188)
>         at
> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:75)
>         at
> org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.handleUpstream(NettyServer.java:173)
>         at
> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
>         at
> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:792)
>         at
> org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296)
>         at
> org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:321)
>         at
> org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:303)
>         at
> org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:208)
>         at
> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:75)
>         at
> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
>         at
> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
>         at
> org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268)
>         at
> org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255)
>         at
> org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:94)
>         at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.processSelectedKeys(AbstractNioWorker.java:364)
>         at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:238)
>         at
> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>         at
> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)