Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Flume >> mail # user >> File Channel error stops flume


Copy link to this message
-
Re: File Channel error stops flume
Thanks Hari. I had waited for 20 minutes and this did not move change. Now, after more than an hour, I see it working
________________________________
 From: Hari Shreedharan <[EMAIL PROTECTED]>
To: [EMAIL PROTECTED]; Rahul Ravindran <[EMAIL PROTECTED]>
Sent: Monday, February 25, 2013 12:46 PM
Subject: Re: File Channel error stops flume
 

Rahul,

Those messages actually just suggest that your channel is replaying. The channel will complete the replay and the agent will start the sinks once the channel is ready. It might take a few minutes based on how many events you have in the channel.
Hari
-- 
Hari Shreedharan

On Monday, February 25, 2013 at 12:07 PM, Rahul Ravindran wrote:
Hi,
>   I modified a parameter to the HDFS sink on a flume config (added an idleInterval) on 2 machines. Things worked fine on one, and not on the other. I tried restarting flume a couple of times and I continue seeing the same log statement (bolded below) with no writes to HDFS
>
>
>25 Feb 2013 08:27:00,174 INFO  [Log-BackgroundWorker-ch2] (org.apache.flume.channel.file.EventQueueBackingStoreFile.checkpoint:109)  - Start checkpoint for /flume2/checkpoint/checkpoint, elements to sync = 8506
>:%
>25 Feb 2013 19:55:51,577 INFO  [lifecycleSupervisor-1-0] (org.apache.flume.channel.file.ReplayHandler.replayLog:236)  - Replaying /flume2/data/log-17
>25 Feb 2013 19:55:51,585 INFO  [lifecycleSupervisor-1-1] (org.apache.flume.channel.file.ReplayHandler.replayLog:236)  - Replaying /flume1/data/log-17
>25 Feb 2013 19:55:51,588 INFO  [lifecycleSupervisor-1-0] (org.apache.flume.tools.DirectMemoryUtils.getDefaultDirectMemorySize:113)  - Unable to get maxDirectMemory from VM: NoSuchMethodException: sun.misc.VM.maxDirectMemory(null)
>25 Feb 2013 19:55:51,592 INFO  [lifecycleSupervisor-1-0] (org.apache.flume.tools.DirectMemoryUtils.allocate:47)  - Direct Memory Allocation:  Allocation = 1048576, Allocated = 0, MaxDirectMemorySize = 268435456, Remaining = 268435456
>25 Feb 2013 19:55:51,634 INFO  [lifecycleSupervisor-1-1] (org.apache.flume.channel.file.LogFile$SequentialReader.skipToLastCheckpointPosition:466)  - fast-forward to checkpoint position: 1622812128
>25 Feb 2013 19:55:51,634 INFO  [lifecycleSupervisor-1-0] (org.apache.flume.channel.file.LogFile$SequentialReader.skipToLastCheckpointPosition:466)  - fast-forward to checkpoint position: 1622720601
>25 Feb 2013 19:55:51,654 INFO  [lifecycleSupervisor-1-0] (org.apache.flume.channel.file.ReplayHandler.replayLog:236)  - Replaying /flume2/data/log-18
>25 Feb 2013 19:55:51,655 INFO  [lifecycleSupervisor-1-0] (org.apache.flume.channel.file.LogFile$SequentialReader.skipToLastCheckpointPosition:466)  - fast-forward to checkpoint position: 1622821593
>25 Feb 2013 19:55:51,655 INFO  [lifecycleSupervisor-1-0] (org.apache.flume.channel.file.ReplayHandler.replayLog:236)  - Replaying /flume2/data/log-19
>25 Feb 2013 19:55:51,656 INFO  [lifecycleSupervisor-1-0] (org.apache.flume.channel.file.LogFile$SequentialReader.skipToLastCheckpointPosition:466)  - fast-forward to checkpoint position: 1622678590
>25 Feb 2013 19:55:51,656 INFO  [lifecycleSupervisor-1-0] (org.apache.flume.channel.file.ReplayHandler.replayLog:236)  - Replaying /flume2/data/log-20
>25 Feb 2013 19:55:51,657 INFO  [lifecycleSupervisor-1-0] (org.apache.flume.channel.file.LogFile$SequentialReader.skipToLastCheckpointPosition:466)  - fast-forward to checkpoint position: 244707334
>25 Feb 2013 19:55:51,657 INFO  [lifecycleSupervisor-1-0] (org.apache.flume.channel.file.ReplayHandler.replayLog:236)  - Replaying /flume2/data/log-21
>25 Feb 2013 19:55:51,657 INFO  [lifecycleSupervisor-1-0] (org.apache.flume.channel.file.LogFile$SequentialReader.skipToLastCheckpointPosition:466)  - fast-forward to checkpoint position: 530601497
>25 Feb 2013 19:55:51,658 INFO  [lifecycleSupervisor-1-0] (org.apache.flume.channel.file.LogFile$SequentialReader.next:491)  - Encountered EOF at 530601497 in /flume2/data/log-21
>25 Feb 2013 19:55:51,658 INFO  [lifecycleSupervisor-1-0] (org.apache.flume.channel.file.ReplayHandler.replayLog:236)  - Replaying /flume2/data/log-22