|
|
-
Take list for MemoryTransaction, capacity 100 full ?周梦想 2013-02-26, 05:57
hello,
I using flume-ng send data from windows to linux hdfs through avro protocol, and encountered this error: 2013-02-26 12:21:02,908 (pool-8-thread-1) [DEBUG - org.apache.flume.source.AvroSource.appendBatch(AvroSource.java:244)] Avro source userlogsrc: Received avro event batch of 100 events. 2013-02-26 12:21:03,107 (SinkRunner-PollingRunner-DefaultSinkProcessor) [ERROR - org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:460)] process failed org.apache.flume.ChannelException: Take list for MemoryTransaction, capacity 100 full, consider committing more frequently, increasing capacity, or increasing thread count at org.apache.flume.channel.MemoryChannel$MemoryTransaction.doTake(MemoryChannel.java:100) at org.apache.flume.channel.BasicTransactionSemantics.take(BasicTransactionSemantics.java:113) at org.apache.flume.channel.BasicChannelSemantics.take(BasicChannelSemantics.java:95) at org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:391) at org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68) at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147) at java.lang.Thread.run(Thread.java:722) Caused by: org.apache.flume.ChannelException: Space for commit to queue couldn't be acquired Sinks are likely not keeping up with sources, or the buffer size is too tight at org.apache.flume.channel.MemoryChannel$MemoryTransaction.doCommit(MemoryChannel.java:126) at org.apache.flume.channel.BasicTransactionSemantics.commit(BasicTransactionSemantics.java:151) at org.apache.flume.channel.ChannelProcessor.processEventBatch(ChannelProcessor.java:192) ... 28 more I have set memory channel capacity to 1000, but it still report this error. some one can give me any advice? Thanks, Andy hdfs.conf: agent46.sources = userlogsrc gamelogsrc agent46.channels = memch1 agent46.sinks = myhdfssink #channels: agent46.channels.memch1.type = memory agent46.channels.memch1.capacity = 10000 agent46.channels.memch1.transactionCapactiy = 100 #sources: #userlogsrc: #agent46.sources.userlogsrc.type = syslogTcp agent46.sources.userlogsrc.type = avro agent46.sources.userlogsrc.port = 5140 agent46.sources.userlogsrc.bind= 0.0.0.0 #agent46.sources.userlogsrc.host= hadoop48 agent46.sources.userlogsrc.interceptors = i1 i2 i3 agent46.sources.userlogsrc.interceptors.i1.type org.apache.flume.interceptor.HostInterceptor$Builder agent46.sources.userlogsrc.interceptors.i1.preserveExisting = true #agent46.sources.userlogsrc.interceptors.i1.hostHeader = hostname agent46.sources.userlogsrc.interceptors.i1.useIP = false agent46.sources.userlogsrc.interceptors.i2.type org.apache.flume.interceptor.TimestampInterceptor$Builder agent46.sources.userlogsrc.interceptors.i3.type = static agent46.sources.userlogsrc.interceptors.i3.key = datacenter agent46.sources.userlogsrc.interceptors.i3.value = userdata agent46.sources.userlogsrc.channels = memch1 #gamelogsrc: #agent46.sources.gamelogsrc.type = syslogTcp agent46.sources.gamelogsrc.type = avro agent46.sources.gamelogsrc.port = 5150 agent46.sources.gamelogsrc.bind= 0.0.0.0 agent46.sources.gamelogsrc.channels = memch1 #sinks: agent46.sinks.myhdfssink.channel = memch1 agent46.sinks.myhdfssink.type = hdfs agent46.sinks.myhdfssink.hdfs.rollInterval = 120 agent46.sinks.myhdfssink.hdfs.appendTimeout = 1000 agent46.sinks.myhdfssink.hdfs.rollSize = 209715200 agent46.sinks.myhdfssink.hdfs.rollCount = 600000 agent46.sinks.myhdfssink.hdfs.batchSize = 1000 agent46.sinks.myhdfssink.hdfs.txnEventMax = 100000 agent46.sinks.myhdfssink.hdfs.threadsPoolSize= 100 agent46.sinks.myhdfssink.hdfs.path = hdfs://h46:9000/flume/%{filename}/%m%d #agent46.sinks.myhdfssink.hdfs.filePrefix = userlogsrc.%{host} #agent46.sinks.myhdfssink.hdfs.filePrefix %{filename}.%{hostname}.%{datacenter}.%Y%m%d agent46.sinks.myhdfssink.hdfs.filePrefix = %{filename}.%{host}.%Y%m%d #agent46.sinks.myhdfssink.hdfs.rollInterval = 60 #agent46.sinks.myhdfssink.hdfs.fileType = SequenceFile agent46.sinks.myhdfssink.hdfs.fileType = DataStream #agent46.sinks.myhdfssink.hdfs.file.writeFormat= Text |