Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Flume >> mail # user >> File Channel error stops flume


Copy link to this message
-
File Channel error stops flume
Hi,
   I modified a parameter to the HDFS sink on a flume config (added an idleInterval) on 2 machines. Things worked fine on one, and not on the other. I tried restarting flume a couple of times and I continue seeing the same log statement (bolded below) with no writes to HDFS

25 Feb 2013 08:27:00,174 INFO  [Log-BackgroundWorker-ch2] (org.apache.flume.channel.file.EventQueueBackingStoreFile.checkpoint:109)  - Start checkpoint for /flume2/checkpoint/checkpoint, elements to sync = 8506
:%
25 Feb 2013 19:55:51,577 INFO  [lifecycleSupervisor-1-0] (org.apache.flume.channel.file.ReplayHandler.replayLog:236)  - Replaying /flume2/data/log-17
25 Feb 2013 19:55:51,585 INFO  [lifecycleSupervisor-1-1] (org.apache.flume.channel.file.ReplayHandler.replayLog:236)  - Replaying /flume1/data/log-17
25 Feb 2013 19:55:51,588 INFO  [lifecycleSupervisor-1-0] (org.apache.flume.tools.DirectMemoryUtils.getDefaultDirectMemorySize:113)  - Unable to get maxDirectMemory from VM: NoSuchMethodException: sun.misc.VM.maxDirectMemory(null)
25 Feb 2013 19:55:51,592 INFO  [lifecycleSupervisor-1-0] (org.apache.flume.tools.DirectMemoryUtils.allocate:47)  - Direct Memory Allocation:  Allocation = 1048576, Allocated = 0, MaxDirectMemorySize = 268435456, Remaining = 268435456
25 Feb 2013 19:55:51,634 INFO  [lifecycleSupervisor-1-1] (org.apache.flume.channel.file.LogFile$SequentialReader.skipToLastCheckpointPosition:466)  - fast-forward to checkpoint position: 1622812128
25 Feb 2013 19:55:51,634 INFO  [lifecycleSupervisor-1-0] (org.apache.flume.channel.file.LogFile$SequentialReader.skipToLastCheckpointPosition:466)  - fast-forward to checkpoint position: 1622720601
25 Feb 2013 19:55:51,654 INFO  [lifecycleSupervisor-1-0] (org.apache.flume.channel.file.ReplayHandler.replayLog:236)  - Replaying /flume2/data/log-18
25 Feb 2013 19:55:51,655 INFO  [lifecycleSupervisor-1-0] (org.apache.flume.channel.file.LogFile$SequentialReader.skipToLastCheckpointPosition:466)  - fast-forward to checkpoint position: 1622821593
25 Feb 2013 19:55:51,655 INFO  [lifecycleSupervisor-1-0] (org.apache.flume.channel.file.ReplayHandler.replayLog:236)  - Replaying /flume2/data/log-19
25 Feb 2013 19:55:51,656 INFO  [lifecycleSupervisor-1-0] (org.apache.flume.channel.file.LogFile$SequentialReader.skipToLastCheckpointPosition:466)  - fast-forward to checkpoint position: 1622678590
25 Feb 2013 19:55:51,656 INFO  [lifecycleSupervisor-1-0] (org.apache.flume.channel.file.ReplayHandler.replayLog:236)  - Replaying /flume2/data/log-20
25 Feb 2013 19:55:51,657 INFO  [lifecycleSupervisor-1-0] (org.apache.flume.channel.file.LogFile$SequentialReader.skipToLastCheckpointPosition:466)  - fast-forward to checkpoint position: 244707334
25 Feb 2013 19:55:51,657 INFO  [lifecycleSupervisor-1-0] (org.apache.flume.channel.file.ReplayHandler.replayLog:236)  - Replaying /flume2/data/log-21
25 Feb 2013 19:55:51,657 INFO  [lifecycleSupervisor-1-0] (org.apache.flume.channel.file.LogFile$SequentialReader.skipToLastCheckpointPosition:466)  - fast-forward to checkpoint position: 530601497
25 Feb 2013 19:55:51,658 INFO  [lifecycleSupervisor-1-0] (org.apache.flume.channel.file.LogFile$SequentialReader.next:491)  - Encountered EOF at 530601497 in /flume2/data/log-21
25 Feb 2013 19:55:51,658 INFO  [lifecycleSupervisor-1-0] (org.apache.flume.channel.file.ReplayHandler.replayLog:236)  - Replaying /flume2/data/log-22
25 Feb 2013 19:55:51,658 INFO  [lifecycleSupervisor-1-1] (org.apache.flume.channel.file.ReplayHandler.replayLog:236)  - Replaying /flume1/data/log-18
25 Feb 2013 19:55:51,658 WARN  [lifecycleSupervisor-1-0] (org.apache.flume.channel.file.LogFile$SequentialReader.skipToLastCheckpointPosition:470)  - Checkpoint for file(/flume2/data/log-22) is: 1361844516782, which is beyond the requested checkpoint time: 1361844516783 and position 0
25 Feb 2013 19:55:51,659 INFO  [lifecycleSupervisor-1-1] (org.apache.flume.channel.file.LogFile$SequentialReader.skipToLastCheckpointPosition:466)  - fast-forward to checkpoint position: 1622674426
25 Feb 2013 19:55:51,659 INFO  [lifecycleSupervisor-1-0] (org.apache.flume.channel.file.ReplayHandler.replayLog:236)  - Replaying /flume2/data/log-23
25 Feb 2013 19:55:51,659 INFO  [lifecycleSupervisor-1-1] (org.apache.flume.channel.file.ReplayHandler.replayLog:236)  - Replaying /flume1/data/log-19
25 Feb 2013 19:55:51,659 WARN  [lifecycleSupervisor-1-0] (org.apache.flume.channel.file.LogFile$SequentialReader.skipToLastCheckpointPosition:470)  - Checkpoint for file(/flume2/data/log-23) is: 1361844516783, which is beyond the requested checkpoint time: 1361844516783 and position 0
25 Feb 2013 19:55:51,660 INFO  [lifecycleSupervisor-1-1] (org.apache.flume.channel.file.LogFile$SequentialReader.skipToLastCheckpointPosition:466)  - fast-forward to checkpoint position: 1622239091
25 Feb 2013 19:55:51,660 INFO  [lifecycleSupervisor-1-1] (org.apache.flume.channel.file.ReplayHandler.replayLog:236)  - Replaying /flume1/data/log-20
25 Feb 2013 19:55:51,661 INFO  [lifecycleSupervisor-1-1] (org.apache.flume.channel.file.LogFile$SequentialReader.skipToLastCheckpointPosition:466)  - fast-forward to checkpoint position: 221490603
25 Feb 2013 19:55:51,661 INFO  [lifecycleSupervisor-1-1] (org.apache.flume.channel.file.ReplayHandler.replayLog:236)  - Replaying /flume1/data/log-21
25 Feb 2013 19:55:51,661 INFO  [lifecycleSupervisor-1-1] (org.apache.flume.channel.file.LogFile$SequentialReader.skipToLastCheckpointPosition:466)  - fast-forward to checkpoint position: 532696754
25 Feb 2013 19:55:52,048 INFO  [lifecycleSupervisor-1-1] (org.apache.flume.channel.file.LogFile$SequentialReader.next:491)  - Encountered EOF at 1623195597 in /flume1/data/log-17
25 Feb 2013 19:55:52,103 INFO  [lifecycleSupervisor-1-0] (org.apache.flume.channel.file.LogFile$SequentialReader.next:491)  - Encountered EOF at 1623195583 in /flume2/data/log-17
25 Feb 2013 19:55:52,308 INFO  [lifecycleSupervisor-1-0] (org.apache.flume.channel.file.LogFile$SequentialReader.next:491)  - Encountered EOF at 1623195536 in
+
Hari Shreedharan 2013-02-25, 20:46
+
Rahul Ravindran 2013-02-25, 21:20
+
Hari Shreedharan 2013-02-25, 21:30
+
Rahul Ravindran 2013-02-26, 00:28
+
Hari Shreedharan 2013-03-01, 22:57