Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Flume >> mail # user >> Take list for MemoryTransaction, capacity 100 full ?


Copy link to this message
-
Take list for MemoryTransaction, capacity 100 full ?
hello,
I using flume-ng send data from windows to linux hdfs through avro
protocol, and encountered this error:

2013-02-26 12:21:02,908 (pool-8-thread-1) [DEBUG -
org.apache.flume.source.AvroSource.appendBatch(AvroSource.java:244)] Avro
source userlogsrc: Received avro event batch of 100 events.

2013-02-26 12:21:03,107 (SinkRunner-PollingRunner-DefaultSinkProcessor)
[ERROR -
org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:460)]
process failed
org.apache.flume.ChannelException: Take list for MemoryTransaction,
capacity 100 full, consider committing more frequently, increasing
capacity, or increasing thread count
        at
org.apache.flume.channel.MemoryChannel$MemoryTransaction.doTake(MemoryChannel.java:100)
        at
org.apache.flume.channel.BasicTransactionSemantics.take(BasicTransactionSemantics.java:113)
        at
org.apache.flume.channel.BasicChannelSemantics.take(BasicChannelSemantics.java:95)
        at
org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:391)
        at
org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68)
        at
org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
        at java.lang.Thread.run(Thread.java:722)
Caused by: org.apache.flume.ChannelException: Space for commit to queue
couldn't be acquired Sinks are likely not keeping up with sources, or the
buffer size is too tight
        at
org.apache.flume.channel.MemoryChannel$MemoryTransaction.doCommit(MemoryChannel.java:126)
        at
org.apache.flume.channel.BasicTransactionSemantics.commit(BasicTransactionSemantics.java:151)
        at
org.apache.flume.channel.ChannelProcessor.processEventBatch(ChannelProcessor.java:192)
        ... 28 more

I have set memory channel capacity to 1000, but it still report this error.
some one can give me any advice?

Thanks,
Andy

hdfs.conf:

agent46.sources = userlogsrc gamelogsrc
agent46.channels = memch1
agent46.sinks = myhdfssink

#channels:
agent46.channels.memch1.type = memory
agent46.channels.memch1.capacity = 10000
agent46.channels.memch1.transactionCapactiy = 100
#sources:
#userlogsrc:
#agent46.sources.userlogsrc.type = syslogTcp
agent46.sources.userlogsrc.type = avro
agent46.sources.userlogsrc.port = 5140
agent46.sources.userlogsrc.bind= 0.0.0.0
#agent46.sources.userlogsrc.host= hadoop48
agent46.sources.userlogsrc.interceptors = i1 i2 i3
agent46.sources.userlogsrc.interceptors.i1.type org.apache.flume.interceptor.HostInterceptor$Builder
agent46.sources.userlogsrc.interceptors.i1.preserveExisting = true
#agent46.sources.userlogsrc.interceptors.i1.hostHeader = hostname
agent46.sources.userlogsrc.interceptors.i1.useIP = false
agent46.sources.userlogsrc.interceptors.i2.type org.apache.flume.interceptor.TimestampInterceptor$Builder
agent46.sources.userlogsrc.interceptors.i3.type = static
agent46.sources.userlogsrc.interceptors.i3.key = datacenter
agent46.sources.userlogsrc.interceptors.i3.value = userdata
agent46.sources.userlogsrc.channels = memch1
#gamelogsrc:
#agent46.sources.gamelogsrc.type = syslogTcp
agent46.sources.gamelogsrc.type = avro
agent46.sources.gamelogsrc.port = 5150
agent46.sources.gamelogsrc.bind= 0.0.0.0
agent46.sources.gamelogsrc.channels = memch1
#sinks:
agent46.sinks.myhdfssink.channel = memch1

agent46.sinks.myhdfssink.type = hdfs
agent46.sinks.myhdfssink.hdfs.rollInterval = 120
agent46.sinks.myhdfssink.hdfs.appendTimeout = 1000
agent46.sinks.myhdfssink.hdfs.rollSize = 209715200
agent46.sinks.myhdfssink.hdfs.rollCount = 600000
agent46.sinks.myhdfssink.hdfs.batchSize = 1000
agent46.sinks.myhdfssink.hdfs.txnEventMax = 100000
agent46.sinks.myhdfssink.hdfs.threadsPoolSize= 100
agent46.sinks.myhdfssink.hdfs.path = hdfs://h46:9000/flume/%{filename}/%m%d
#agent46.sinks.myhdfssink.hdfs.filePrefix = userlogsrc.%{host}
#agent46.sinks.myhdfssink.hdfs.filePrefix %{filename}.%{hostname}.%{datacenter}.%Y%m%d
agent46.sinks.myhdfssink.hdfs.filePrefix = %{filename}.%{host}.%Y%m%d
#agent46.sinks.myhdfssink.hdfs.rollInterval = 60
#agent46.sinks.myhdfssink.hdfs.fileType = SequenceFile
agent46.sinks.myhdfssink.hdfs.fileType = DataStream
#agent46.sinks.myhdfssink.hdfs.file.writeFormat= Text
+
周梦想 2013-02-26, 06:45
+
周梦想 2013-02-26, 07:41
+
Hari Shreedharan 2013-02-26, 07:56
+
周梦想 2013-02-27, 02:29
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB