Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Flume, mail # user - flume-ng Unable to put batch on required channel error


Copy link to this message
-
Re: flume-ng Unable to put batch on required channel error
周梦想 2013-03-01, 09:54
thank you. I once increased hdfs batch size more than 1000, but it report
the ChannelException more frequence. so I changed small and small, batch
size to 50 is less error.

2013/3/1 Hari Shreedharan <[EMAIL PROTECTED]>

>  Try increasing the callTimeout for HDFS Sink - that is in milliseconds.
> Also, you should try using a batch size/txnEvent max of at least 100 (or
> more). Using more HDFS sinks will also improve performance by increased
> parallel writes  since each sink is single threaded and writes to only one
> file at a time. (note that each HDFS Sink should write to a different
> directory).
>
> --
> Hari Shreedharan
>
> On Thursday, February 28, 2013 at 7:10 PM, 周梦想 wrote:
>
> Hello,
>
> I wrote a test java file to put local files to hdfs, it put 1.1GB files to
> hdfs spent about 35 seconds. while using flume, 100MB files it should spend
> about 3-5 minutes.
> why flume is so slow and always report error? how to enlarge the flume
> hdfs sink throughput?
>
> Andy
>
> 2013/3/1 周梦想 <[EMAIL PROTECTED]>
>
> hello,
> I saw lot's of flume-ng 1.3.1 agent Exception said  Unable to put batch on
> required channel,and then disconnected.
> what is the effect? lost data or duplicate data?
> What caused this error?
> How to avoid this error?
>
> some of my conf:
> agent46.channels.memch1.type = memory
> agent46.channels.memch1.capacity = 100000
> agent46.channels.memch1.transactionCapactiy = 1000
>
> agent46.sources.userlogsrc.type = avro
>
> agent46.sinks.myhdfssink.type = hdfs
> agent46.sinks.myhdfssink.hdfs.rollInterval = 620
> agent46.sinks.myhdfssink.hdfs.appendTimeout = 1000
> agent46.sinks.myhdfssink.hdfs.rollSize = 209715200
> agent46.sinks.myhdfssink.hdfs.rollCount = 600000
> agent46.sinks.myhdfssink.hdfs.batchSize = 50
> agent46.sinks.myhdfssink.hdfs.txnEventMax = 100000
> agent46.sinks.myhdfssink.hdfs.threadsPoolSize= 100
>
> ...
> 01 Mar 2013 03:00:16,368 ERROR [pool-7-thread-3]
> (org.apache.flume.source.AvroSource.appendBatch:261)  - Avro source
> gamelogsrc: Unable to process event batch. Exception follows.
> org.apache.flume.ChannelException: Unable to put batch on required
> channel: org.apache.flume.channel.MemoryChannel{name: memch1}
>         at
> org.apache.flume.channel.ChannelProcessor.processEventBatch(ChannelProcessor.java:200)
>         at
> org.apache.flume.source.AvroSource.appendBatch(AvroSource.java:259)
>         at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source)
>         at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>         at java.lang.reflect.Method.invoke(Method.java:601)
>         at
> org.apache.avro.ipc.specific.SpecificResponder.respond(SpecificResponder.java:88)
>         at org.apache.avro.ipc.Responder.respond(Responder.java:149)
>         at
> org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.messageReceived(NettyServer.java:188)
>         at
> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:75)
>         at
> org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.handleUpstream(NettyServer.java:173)
>         at
> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
>         at
> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:792)
>         at
> org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296)
>         at
> org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:321)
>         at
> org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:303)
>         at
> org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:208)
>         at
> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:75)
>         at
> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
>         at