Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Flume, mail # user - Take list for MemoryTransaction, capacity 100 full ?


Copy link to this message
-
Take list for MemoryTransaction, capacity 100 full ?
周梦想 2013-02-26, 05:57
hello,
I using flume-ng send data from windows to linux hdfs through avro
protocol, and encountered this error:

2013-02-26 12:21:02,908 (pool-8-thread-1) [DEBUG -
org.apache.flume.source.AvroSource.appendBatch(AvroSource.java:244)] Avro
source userlogsrc: Received avro event batch of 100 events.

2013-02-26 12:21:03,107 (SinkRunner-PollingRunner-DefaultSinkProcessor)
[ERROR -
org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:460)]
process failed
org.apache.flume.ChannelException: Take list for MemoryTransaction,
capacity 100 full, consider committing more frequently, increasing
capacity, or increasing thread count
        at
org.apache.flume.channel.MemoryChannel$MemoryTransaction.doTake(MemoryChannel.java:100)
        at
org.apache.flume.channel.BasicTransactionSemantics.take(BasicTransactionSemantics.java:113)
        at
org.apache.flume.channel.BasicChannelSemantics.take(BasicChannelSemantics.java:95)
        at
org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:391)
        at
org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68)
        at
org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
        at java.lang.Thread.run(Thread.java:722)
Caused by: org.apache.flume.ChannelException: Space for commit to queue
couldn't be acquired Sinks are likely not keeping up with sources, or the
buffer size is too tight
        at
org.apache.flume.channel.MemoryChannel$MemoryTransaction.doCommit(MemoryChannel.java:126)
        at
org.apache.flume.channel.BasicTransactionSemantics.commit(BasicTransactionSemantics.java:151)
        at
org.apache.flume.channel.ChannelProcessor.processEventBatch(ChannelProcessor.java:192)
        ... 28 more

I have set memory channel capacity to 1000, but it still report this error.
some one can give me any advice?

Thanks,
Andy

hdfs.conf:

agent46.sources = userlogsrc gamelogsrc
agent46.channels = memch1
agent46.sinks = myhdfssink

#channels:
agent46.channels.memch1.type = memory
agent46.channels.memch1.capacity = 10000
agent46.channels.memch1.transactionCapactiy = 100
#sources:
#userlogsrc:
#agent46.sources.userlogsrc.type = syslogTcp
agent46.sources.userlogsrc.type = avro
agent46.sources.userlogsrc.port = 5140
agent46.sources.userlogsrc.bind= 0.0.0.0
#agent46.sources.userlogsrc.host= hadoop48
agent46.sources.userlogsrc.interceptors = i1 i2 i3
agent46.sources.userlogsrc.interceptors.i1.type org.apache.flume.interceptor.HostInterceptor$Builder
agent46.sources.userlogsrc.interceptors.i1.preserveExisting = true
#agent46.sources.userlogsrc.interceptors.i1.hostHeader = hostname
agent46.sources.userlogsrc.interceptors.i1.useIP = false
agent46.sources.userlogsrc.interceptors.i2.type org.apache.flume.interceptor.TimestampInterceptor$Builder
agent46.sources.userlogsrc.interceptors.i3.type = static
agent46.sources.userlogsrc.interceptors.i3.key = datacenter
agent46.sources.userlogsrc.interceptors.i3.value = userdata
agent46.sources.userlogsrc.channels = memch1
#gamelogsrc:
#agent46.sources.gamelogsrc.type = syslogTcp
agent46.sources.gamelogsrc.type = avro
agent46.sources.gamelogsrc.port = 5150
agent46.sources.gamelogsrc.bind= 0.0.0.0
agent46.sources.gamelogsrc.channels = memch1
#sinks:
agent46.sinks.myhdfssink.channel = memch1

agent46.sinks.myhdfssink.type = hdfs
agent46.sinks.myhdfssink.hdfs.rollInterval = 120
agent46.sinks.myhdfssink.hdfs.appendTimeout = 1000
agent46.sinks.myhdfssink.hdfs.rollSize = 209715200
agent46.sinks.myhdfssink.hdfs.rollCount = 600000
agent46.sinks.myhdfssink.hdfs.batchSize = 1000
agent46.sinks.myhdfssink.hdfs.txnEventMax = 100000
agent46.sinks.myhdfssink.hdfs.threadsPoolSize= 100
agent46.sinks.myhdfssink.hdfs.path = hdfs://h46:9000/flume/%{filename}/%m%d
#agent46.sinks.myhdfssink.hdfs.filePrefix = userlogsrc.%{host}
#agent46.sinks.myhdfssink.hdfs.filePrefix %{filename}.%{hostname}.%{datacenter}.%Y%m%d
agent46.sinks.myhdfssink.hdfs.filePrefix = %{filename}.%{host}.%Y%m%d
#agent46.sinks.myhdfssink.hdfs.rollInterval = 60
#agent46.sinks.myhdfssink.hdfs.fileType = SequenceFile
agent46.sinks.myhdfssink.hdfs.fileType = DataStream
#agent46.sinks.myhdfssink.hdfs.file.writeFormat= Text