Flume, mail # user - Re: 回复： About file channel yum error

Re: 回复： About file channel yum error

Brock Noland 2012-10-16, 15:10

When the agent sees an OOM, can you take a thread dump (kill -3 <pid>) and
post it on pastebin? Please be sure to include the heap summary section at
the bottom of the dump.

Brock

On Mon, Oct 15, 2012 at 10:08 PM, Yanzhi.liu <[EMAIL PROTECTED]> wrote:

> Hello Brock:
> My flume's configuration is
> agent_foo.sources = tailsource-1
> agent_foo.channels = fileChannel
> agent_foo.sinks = hdfsSink-1
> agent_foo.sources.tailsource-1.type > com.chinacache.cpis.sources.CpisAvroSource
> agent_foo.sources.tailsource-1.channels=fileChannel
> agent_foo.sources.tailsource-1.bind=221.130.18.90
> agent_foo.sources.tailsource-1.port=4545
> #agent_foo.sources.tailsource-1.threads=10
> agent_foo.channels.fileChannel.type = file
> agent_foo.channels.fileChannel.checkpointDir=/home/hadoop/checkpoint
> agent_foo.channels.fileChannel.dataDirs=/data/cache2/data
> agent_foo.channels.fileChannel.maxFileSize=524288000
> #agent_foo.channels.fileChannel.capacity = 10000
> agent_foo.sinks.hdfsSink-1.type = hdfs
> agent_foo.sinks.hdfsSink-1.channel = fileChannel
> agent_foo.sinks.hdfsSink-1.hdfs.path > hdfs://CMN-NJ-2-579:9000/user/hadoop/fc_logs/%{month}/%{day}/%{deviceId}/%{hour}
> #agent_foo.sinks.hdfsSink-1.hdfs.path > hdfs://CMN-NJ-2-579:9000/user/hadoop/fc_logs/%{month}/%{day}/%{hour}/%{deviceId}
> agent_foo.sinks.hdfsSink-1.hdfs.filePrefix = cpisfc-assess.log
> agent_foo.sinks.hdfsSink-1.hdfs.rollInterval=600
> agent_foo.sinks.hdfsSink-1.hdfs.rollCount=0
> agent_foo.sinks.hdfsSink-1.hdfs.rollSize = 506870912
> agent_foo.sinks.hdfsSink-1.hdfs.fileType=CompressedStream
> agent_foo.sinks.hdfsSink-1.hdfs.codeC=gzip
> agent_foo.sinks.hdfsSink-1.hdfs.writeFormat=Text
> agent_foo.sinks.hdfsSink-1.hdfs.batchSize=1
> agent_foo.sinks.hdfsSink-1.serializer=avro_event
> And the java configuration was JAVA_OPTS="-Xms100m -Xmx2048m
> -Dcom.sun.management.jmxremote";
> Thank you very much!
> My Name :
> Yanzhi Liu
> **
>
>
> ------------------ 原始邮件 ------------------
> *发件人:* "Brock Noland"<[EMAIL PROTECTED]>;
> *发送时间:* 2012年10月16日(星期二) 中午11:01
> *收件人:* "user"<[EMAIL PROTECTED]>; **
> *主题:* Re: About file channel yum error
>
> What is your channel capacity and configured agent heap size?
>
> On Monday, October 15, 2012 at 9:19 PM, Yanzhi.liu wrote:
>
> Hello everyone:
> I am using the flume 1.3.0.But the flume clusters was working after
> two days and the flume was stoped. And the flume.log was only error about
> the:
> 14 Oct 2012 15:29:17,018 ERROR
> [SinkRunner-PollingRunner-DefaultSinkProcessor]
> (org.apache.flume.sink.hdfs.HDFSEventSink.process:448) - process failed
> java.lang.OutOfMemoryError
> at java.io.RandomAccessFile.readBytes(Native Method)
> at java.io.RandomAccessFile.read(RandomAccessFile.java:338)
> at java.io.RandomAccessFile.readFully(RandomAccessFile.java:397)
> at java.io.RandomAccessFile.readFully(RandomAccessFile.java:377)
> at
> org.apache.flume.channel.file.LogFile.readDelimitedBuffer(LogFile.java:486)
> at
> org.apache.flume.channel.file.LogFileV3$RandomReader.doGet(LogFileV3.java:258)
> at
> org.apache.flume.channel.file.LogFile$RandomReader.get(LogFile.java:298)
> at org.apache.flume.channel.file.Log.get(Log.java:409)
> at
> org.apache.flume.channel.file.FileChannel$FileBackedTransaction.doTake(FileChannel.java:447)
> at
> org.apache.flume.channel.BasicTransactionSemantics.take(BasicTransactionSemantics.java:113)
> at
> org.apache.flume.channel.BasicChannelSemantics.take(BasicChannelSemantics.java:91)
> at
> org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:387)
> at
> org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68)
> at
> org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
> at java.lang.Thread.run(Thread.java:662)
> So I want to how to avoid the error.
> Thanks very much!
> My Name:
> Yanzhi Liu
>
>
> **
>

--
Apache MRUnit - Unit testing MapReduce - http://incubator.apache.org/mrunit/

Re: 回复： About file channel yum error

Kevin Lee 2012-10-23, 02:26

Hey,

I got a OOM error too, and the detail as follow:

USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
flume 22902 59.6 23.5 2813560 1802672 ? Sl Oct18 4210:34 /usr/java/jdk1.6.0_33/bin/java -Xms500m -Xmx1500m -cp /etc/flume-ng/conf:/usr/lib/flume-ng/lib/*:/etc/hadoop/conf:/usr/lib/hadoop/lib/activation-1.1.jar

free -m
total used free shared buffers cached
Mem: 7467 7430 36 0 13 5011
-/+ buffers/cache: 2405 5061
Swap: 4095 15 4080

thread dump:

/usr/java/default/bin/jstack -F 22902
Attaching to process ID 22902, please wait...
Debugger attached successfully.
Server compiler detected.
JVM version is 20.8-b03
Deadlock Detection:

No deadlocks found.

Thread 11929: (state = BLOCKED)
- sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
- java.util.concurrent.locks.LockSupport.parkNanos(java.lang.Object, long) @bci=20, line=196 (Compiled frame)
- java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(long) @bci=68, line=2025 (Compiled frame)
- java.util.concurrent.DelayQueue.take() @bci=57, line=164 (Compiled frame)
- java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take() @bci=4, line=609 (Compiled frame)
- java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take() @bci=1, line=602 (Compiled frame)
- java.util.concurrent.ThreadPoolExecutor.getTask() @bci=78, line=947 (Compiled frame)
- java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=18, line=907 (Compiled frame)
- java.lang.Thread.run() @bci=11, line=662 (Interpreted frame)
Thread 23734: (state = BLOCKED)
- sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
- java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, line=156 (Compiled frame)
- java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await() @bci=42, line=1987 (Compiled frame)
- java.util.concurrent.LinkedBlockingQueue.take() @bci=29, line=399 (Compiled frame)
- java.util.concurrent.ThreadPoolExecutor.getTask() @bci=78, line=947 (Compiled frame)
- java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=18, line=907 (Compiled frame)
- java.lang.Thread.run() @bci=11, line=662 (Interpreted frame)
Thread 23733: (state = BLOCKED)
- sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
- java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, line=156 (Compiled frame)
- java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await() @bci=42, line=1987 (Compiled frame)
- java.util.concurrent.LinkedBlockingQueue.take() @bci=29, line=399 (Compiled frame)
- java.util.concurrent.ThreadPoolExecutor.getTask() @bci=78, line=947 (Compiled frame)
- java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=18, line=907 (Compiled frame)
- java.lang.Thread.run() @bci=11, line=662 (Interpreted frame)
Thread 23732: (state = BLOCKED)
- sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
- java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, line=156 (Compiled frame)
- java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await() @bci=42, line=1987 (Compiled frame)
- java.util.concurrent.LinkedBlockingQueue.take() @bci=29, line=399 (Compiled frame)
- java.util.concurrent.ThreadPoolExecutor.getTask() @bci=78, line=947 (Compiled frame)
- java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=18, line=907 (Compiled frame)
- java.lang.Thread.run() @bci=11, line=662 (Interpreted frame)
Thread 23731: (state = BLOCKED)
- sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
- java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, line=156 (Compiled frame)
- java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await() @bci=42, line=1987 (Compiled frame)
- java.util.concurrent.LinkedBlockingQueue.take() @bci=29, line=399 (Compiled frame)
- java.util.concurrent.ThreadPoolExecutor.getTask() @bci=78, line=947 (Compiled frame)
- java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=18, line=907 (Compiled frame)
- java.lang.Thread.run() @bci=11, line=662 (Interpreted frame)
Thread 23730: (state = BLOCKED)
- sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
- java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, line=156 (Compiled frame)
- java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await() @bci=42, line=1987 (Compiled frame)
- java.util.concurrent.LinkedBlockingQueue.take() @bci=29, line=399 (Compiled frame)
- java.util.concurrent.ThreadPoolExecutor.getTask() @bci=78, line=947 (Compiled frame)
- java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=18, line=907 (Compiled frame)
- java.lang.Thread.run() @bci=11, line=662 (Interpreted frame)
Thread 23729: (state = BLOCKED)
- sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
- java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, line=156 (Compiled frame)
- java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await() @bci=42, line=1987 (Compiled frame)
- java.util.concurrent.LinkedBlockingQueue.take() @bci=29, line=399 (Compiled frame)
- java.util.concurrent.ThreadPoolExecutor.getTask() @bci=78, line=947 (Compiled frame)
- java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=18, line=907 (Compiled frame)
- java.lang.Thread.run() @bci=11, line=662 (Interpreted frame)
Thread 23728: (state = BLOCKED)
- sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
- java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, line=156 (Compiled frame)
- java.util.concurrent

Re: 回复： About file channel yum error

Brock Noland 2012-10-23, 02:31

Hi,

Sorry that thread dump doesn't provide much. If you use kill -3 <pid> it
prints heap information at the bottom of the of the dump. Also the OOM
error message long with a configuration would be helpful.

Brock

On Mon, Oct 22, 2012 at 9:26 PM, Kevin Lee <[EMAIL PROTECTED]> wrote:

> Hey,
>
> I got a OOM error too, and the detail as follow:
>
> USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
> flume 22902 59.6 23.5 2813560 1802672 ? Sl Oct18 4210:34
> /usr/java/jdk1.6.0_33/bin/java -Xms500m -Xmx1500m -cp
> /etc/flume-ng/conf:/usr/lib/flume-ng/lib/*:/etc/hadoop/conf:/usr/lib/hadoop/lib/activation-1.1.jar
>
> free -m
> total used free shared buffers cached
> Mem: 7467 7430 36 0 13 5011
> -/+ buffers/cache: 2405 5061
> Swap: 4095 15 4080
>
> thread dump:
>
> /usr/java/default/bin/jstack -F 22902
> Attaching to process ID 22902, please wait...
> Debugger attached successfully.
> Server compiler detected.
> JVM version is 20.8-b03
> Deadlock Detection:
>
> No deadlocks found.
>
> Thread 11929: (state = BLOCKED)
> - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information
> may be imprecise)
> - java.util.concurrent.locks.LockSupport.parkNanos(java.lang.Object,
> long) @bci=20, line=196 (Compiled frame)
> -
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(long)
> @bci=68, line=2025 (Compiled frame)
> - java.util.concurrent.DelayQueue.take() @bci=57, line=164 (Compiled
> frame)
> -
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take()
> @bci=4, line=609 (Compiled frame)
> -
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take()
> @bci=1, line=602 (Compiled frame)
> - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=78, line=947
> (Compiled frame)
> - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=18, line=907
> (Compiled frame)
> - java.lang.Thread.run() @bci=11, line=662 (Interpreted frame)
>
>
> Thread 23734: (state = BLOCKED)
> - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information
> may be imprecise)
> - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14,
> line=156 (Compiled frame)
> -
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await()
> @bci=42, line=1987 (Compiled frame)
> - java.util.concurrent.LinkedBlockingQueue.take() @bci=29, line=399
> (Compiled frame)
> - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=78, line=947
> (Compiled frame)
> - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=18, line=907
> (Compiled frame)
> - java.lang.Thread.run() @bci=11, line=662 (Interpreted frame)
>
>
> Thread 23733: (state = BLOCKED)
> - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information
> may be imprecise)
> - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14,
> line=156 (Compiled frame)
> -
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await()
> @bci=42, line=1987 (Compiled frame)
> - java.util.concurrent.LinkedBlockingQueue.take() @bci=29, line=399
> (Compiled frame)
> - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=78, line=947
> (Compiled frame)
> - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=18, line=907
> (Compiled frame)
> - java.lang.Thread.run() @bci=11, line=662 (Interpreted frame)
>
>
> Thread 23732: (state = BLOCKED)
> - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information
> may be imprecise)
> - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14,
> line=156 (Compiled frame)
> -
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await()
> @bci=42, line=1987 (Compiled frame)
> - java.util.concurrent.LinkedBlockingQueue.take() @bci=29, line=399
> (Compiled frame)
> - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=78, line=947
� 中午11:01
Apache MRUnit - Unit testing MapReduce - http://incubator.apache.org/mrunit/

Re: 回复： About file channel yum error

Kevin Lee 2012-10-23, 03:17

Hi,

Today Amazon has EBS outage, maybe it's a not flume issue, thanks you.

-Kevin

On Oct 23, 2012, at 10:31 AM, Brock Noland wrote:

> Hi,
>
> Sorry that thread dump doesn't provide much. If you use kill -3 <pid> it prints heap information at the bottom of the of the dump. Also the OOM error message long with a configuration would be helpful.
>
> Brock
>
> On Mon, Oct 22, 2012 at 9:26 PM, Kevin Lee <[EMAIL PROTECTED]> wrote:
> Hey,
>
> I got a OOM error too, and the detail as follow:
>
> USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
> flume 22902 59.6 23.5 2813560 1802672 ? Sl Oct18 4210:34 /usr/java/jdk1.6.0_33/bin/java -Xms500m -Xmx1500m -cp /etc/flume-ng/conf:/usr/lib/flume-ng/lib/*:/etc/hadoop/conf:/usr/lib/hadoop/lib/activation-1.1.jar
>
> free -m
> total used free shared buffers cached
> Mem: 7467 7430 36 0 13 5011
> -/+ buffers/cache: 2405 5061
> Swap: 4095 15 4080
>
> thread dump:
>
> /usr/java/default/bin/jstack -F 22902
> Attaching to process ID 22902, please wait...
> Debugger attached successfully.
> Server compiler detected.
> JVM version is 20.8-b03
> Deadlock Detection:
>
> No deadlocks found.
>
> Thread 11929: (state = BLOCKED)
> - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
> - java.util.concurrent.locks.LockSupport.parkNanos(java.lang.Object, long) @bci=20, line=196 (Compiled frame)
> - java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(long) @bci=68, line=2025 (Compiled frame)
> - java.util.concurrent.DelayQueue.take() @bci=57, line=164 (Compiled frame)
> - java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take() @bci=4, line=609 (Compiled frame)
> - java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take() @bci=1, line=602 (Compiled frame)
> - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=78, line=947 (Compiled frame)
> - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=18, line=907 (Compiled frame)
> - java.lang.Thread.run() @bci=11, line=662 (Interpreted frame)
>
>
> Thread 23734: (state = BLOCKED)
> - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
> - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, line=156 (Compiled frame)
> - java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await() @bci=42, line=1987 (Compiled frame)
> - java.util.concurrent.LinkedBlockingQueue.take() @bci=29, line=399 (Compiled frame)
> - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=78, line=947 (Compiled frame)
> - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=18, line=907 (Compiled frame)
> - java.lang.Thread.run() @bci=11, line=662 (Interpreted frame)
>
>
> Thread 23733: (state = BLOCKED)
> - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
> - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, line=156 (Compiled frame)
> - java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await() @bci=42, line=1987 (Compiled frame)
> - java.util.concurrent.LinkedBlockingQueue.take() @bci=29, line=399 (Compiled frame)
> - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=78, line=947 (Compiled frame)
> - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=18, line=907 (Compiled frame)
> - java.lang.Thread.run() @bci=11, line=662 (Interpreted frame)
>
>
> Thread 23732: (state = BLOCKED)
> - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
> - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, line=156 (Compiled frame)
> - java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await() @bci=42, line=1987 (Compiled frame)
> - java.util.concurrent.LinkedBlockingQueue.take() @bci=29, line=399 (Compiled frame)