|
|
-
File Channel error stops flume
Rahul Ravindran 2013-02-25, 20:07
Hi,
I modified a parameter to the HDFS sink on a flume config (added an idleInterval) on 2 machines. Things worked fine on one, and not on the other. I tried restarting flume a couple of times and I continue seeing the same log statement (bolded below) with no writes to HDFS
25 Feb 2013 08:27:00,174 INFO [Log-BackgroundWorker-ch2] (org.apache.flume.channel.file.EventQueueBackingStoreFile.checkpoint:109) - Start checkpoint for /flume2/checkpoint/checkpoint, elements to sync = 8506
:%
25 Feb 2013 19:55:51,577 INFO [lifecycleSupervisor-1-0] (org.apache.flume.channel.file.ReplayHandler.replayLog:236) - Replaying /flume2/data/log-17
25 Feb 2013 19:55:51,585 INFO [lifecycleSupervisor-1-1] (org.apache.flume.channel.file.ReplayHandler.replayLog:236) - Replaying /flume1/data/log-17
25 Feb 2013 19:55:51,588 INFO [lifecycleSupervisor-1-0] (org.apache.flume.tools.DirectMemoryUtils.getDefaultDirectMemorySize:113) - Unable to get maxDirectMemory from VM: NoSuchMethodException: sun.misc.VM.maxDirectMemory(null)
25 Feb 2013 19:55:51,592 INFO [lifecycleSupervisor-1-0] (org.apache.flume.tools.DirectMemoryUtils.allocate:47) - Direct Memory Allocation: Allocation = 1048576, Allocated = 0, MaxDirectMemorySize = 268435456, Remaining = 268435456
25 Feb 2013 19:55:51,634 INFO [lifecycleSupervisor-1-1] (org.apache.flume.channel.file.LogFile$SequentialReader.skipToLastCheckpointPosition:466) - fast-forward to checkpoint position: 1622812128
25 Feb 2013 19:55:51,634 INFO [lifecycleSupervisor-1-0] (org.apache.flume.channel.file.LogFile$SequentialReader.skipToLastCheckpointPosition:466) - fast-forward to checkpoint position: 1622720601
25 Feb 2013 19:55:51,654 INFO [lifecycleSupervisor-1-0] (org.apache.flume.channel.file.ReplayHandler.replayLog:236) - Replaying /flume2/data/log-18
25 Feb 2013 19:55:51,655 INFO [lifecycleSupervisor-1-0] (org.apache.flume.channel.file.LogFile$SequentialReader.skipToLastCheckpointPosition:466) - fast-forward to checkpoint position: 1622821593
25 Feb 2013 19:55:51,655 INFO [lifecycleSupervisor-1-0] (org.apache.flume.channel.file.ReplayHandler.replayLog:236) - Replaying /flume2/data/log-19
25 Feb 2013 19:55:51,656 INFO [lifecycleSupervisor-1-0] (org.apache.flume.channel.file.LogFile$SequentialReader.skipToLastCheckpointPosition:466) - fast-forward to checkpoint position: 1622678590
25 Feb 2013 19:55:51,656 INFO [lifecycleSupervisor-1-0] (org.apache.flume.channel.file.ReplayHandler.replayLog:236) - Replaying /flume2/data/log-20
25 Feb 2013 19:55:51,657 INFO [lifecycleSupervisor-1-0] (org.apache.flume.channel.file.LogFile$SequentialReader.skipToLastCheckpointPosition:466) - fast-forward to checkpoint position: 244707334
25 Feb 2013 19:55:51,657 INFO [lifecycleSupervisor-1-0] (org.apache.flume.channel.file.ReplayHandler.replayLog:236) - Replaying /flume2/data/log-21
25 Feb 2013 19:55:51,657 INFO [lifecycleSupervisor-1-0] (org.apache.flume.channel.file.LogFile$SequentialReader.skipToLastCheckpointPosition:466) - fast-forward to checkpoint position: 530601497
25 Feb 2013 19:55:51,658 INFO [lifecycleSupervisor-1-0] (org.apache.flume.channel.file.LogFile$SequentialReader.next:491) - Encountered EOF at 530601497 in /flume2/data/log-21
25 Feb 2013 19:55:51,658 INFO [lifecycleSupervisor-1-0] (org.apache.flume.channel.file.ReplayHandler.replayLog:236) - Replaying /flume2/data/log-22
25 Feb 2013 19:55:51,658 INFO [lifecycleSupervisor-1-1] (org.apache.flume.channel.file.ReplayHandler.replayLog:236) - Replaying /flume1/data/log-18
25 Feb 2013 19:55:51,658 WARN [lifecycleSupervisor-1-0] (org.apache.flume.channel.file.LogFile$SequentialReader.skipToLastCheckpointPosition:470) - Checkpoint for file(/flume2/data/log-22) is: 1361844516782, which is beyond the requested checkpoint time: 1361844516783 and position 0
25 Feb 2013 19:55:51,659 INFO [lifecycleSupervisor-1-1] (org.apache.flume.channel.file.LogFile$SequentialReader.skipToLastCheckpointPosition:466) - fast-forward to checkpoint position: 1622674426
25 Feb 2013 19:55:51,659 INFO [lifecycleSupervisor-1-0] (org.apache.flume.channel.file.ReplayHandler.replayLog:236) - Replaying /flume2/data/log-23
25 Feb 2013 19:55:51,659 INFO [lifecycleSupervisor-1-1] (org.apache.flume.channel.file.ReplayHandler.replayLog:236) - Replaying /flume1/data/log-19
25 Feb 2013 19:55:51,659 WARN [lifecycleSupervisor-1-0] (org.apache.flume.channel.file.LogFile$SequentialReader.skipToLastCheckpointPosition:470) - Checkpoint for file(/flume2/data/log-23) is: 1361844516783, which is beyond the requested checkpoint time: 1361844516783 and position 0
25 Feb 2013 19:55:51,660 INFO [lifecycleSupervisor-1-1] (org.apache.flume.channel.file.LogFile$SequentialReader.skipToLastCheckpointPosition:466) - fast-forward to checkpoint position: 1622239091
25 Feb 2013 19:55:51,660 INFO [lifecycleSupervisor-1-1] (org.apache.flume.channel.file.ReplayHandler.replayLog:236) - Replaying /flume1/data/log-20
25 Feb 2013 19:55:51,661 INFO [lifecycleSupervisor-1-1] (org.apache.flume.channel.file.LogFile$SequentialReader.skipToLastCheckpointPosition:466) - fast-forward to checkpoint position: 221490603
25 Feb 2013 19:55:51,661 INFO [lifecycleSupervisor-1-1] (org.apache.flume.channel.file.ReplayHandler.replayLog:236) - Replaying /flume1/data/log-21
25 Feb 2013 19:55:51,661 INFO [lifecycleSupervisor-1-1] (org.apache.flume.channel.file.LogFile$SequentialReader.skipToLastCheckpointPosition:466) - fast-forward to checkpoint position: 532696754
25 Feb 2013 19:55:52,048 INFO [lifecycleSupervisor-1-1] (org.apache.flume.channel.file.LogFile$SequentialReader.next:491) - Encountered EOF at 1623195597 in /flume1/data/log-17
25 Feb 2013 19:55:52,103 INFO [lifecycleSupervisor-1-0] (org.apache.flume.channel.file.LogFile$SequentialReader.next:491) - Encountered EOF at 1623195583 in /flume2/data/log-17
25 Feb 2013 19:55:52,308 INFO [lifecycleSupervisor-1-0] (org.apache.flume.channel.file.LogFile$SequentialReader.next:491) - Encountered EOF at 1623195536 in
-
Re: File Channel error stops flume
Hari Shreedharan 2013-02-25, 20:46
Rahul,
Those messages actually just suggest that your channel is replaying. The channel will complete the replay and the agent will start the sinks once the channel is ready. It might take a few minutes based on how many events you have in the channel.
Hari
--
Hari Shreedharan
On Monday, February 25, 2013 at 12:07 PM, Rahul Ravindran wrote:
> Hi,
> I modified a parameter to the HDFS sink on a flume config (added an idleInterval) on 2 machines. Things worked fine on one, and not on the other. I tried restarting flume a couple of times and I continue seeing the same log statement (bolded below) with no writes to HDFS
>
> 25 Feb 2013 08:27:00,174 INFO [Log-BackgroundWorker-ch2] (org.apache.flume.channel.file.EventQueueBackingStoreFile.checkpoint:109) - Start checkpoint for /flume2/checkpoint/checkpoint, elements to sync = 8506
> :%
> 25 Feb 2013 19:55:51,577 INFO [lifecycleSupervisor-1-0] (org.apache.flume.channel.file.ReplayHandler.replayLog:236) - Replaying /flume2/data/log-17
> 25 Feb 2013 19:55:51,585 INFO [lifecycleSupervisor-1-1] (org.apache.flume.channel.file.ReplayHandler.replayLog:236) - Replaying /flume1/data/log-17
> 25 Feb 2013 19:55:51,588 INFO [lifecycleSupervisor-1-0] (org.apache.flume.tools.DirectMemoryUtils.getDefaultDirectMemorySize:113) - Unable to get maxDirectMemory from VM: NoSuchMethodException: sun.misc.VM.maxDirectMemory(null)
> 25 Feb 2013 19:55:51,592 INFO [lifecycleSupervisor-1-0] (org.apache.flume.tools.DirectMemoryUtils.allocate:47) - Direct Memory Allocation: Allocation = 1048576, Allocated = 0, MaxDirectMemorySize = 268435456, Remaining = 268435456
> 25 Feb 2013 19:55:51,634 INFO [lifecycleSupervisor-1-1] (org.apache.flume.channel.file.LogFile$SequentialReader.skipToLastCheckpointPosition:466) - fast-forward to checkpoint position: 1622812128
> 25 Feb 2013 19:55:51,634 INFO [lifecycleSupervisor-1-0] (org.apache.flume.channel.file.LogFile$SequentialReader.skipToLastCheckpointPosition:466) - fast-forward to checkpoint position: 1622720601
> 25 Feb 2013 19:55:51,654 INFO [lifecycleSupervisor-1-0] (org.apache.flume.channel.file.ReplayHandler.replayLog:236) - Replaying /flume2/data/log-18
> 25 Feb 2013 19:55:51,655 INFO [lifecycleSupervisor-1-0] (org.apache.flume.channel.file.LogFile$SequentialReader.skipToLastCheckpointPosition:466) - fast-forward to checkpoint position: 1622821593
> 25 Feb 2013 19:55:51,655 INFO [lifecycleSupervisor-1-0] (org.apache.flume.channel.file.ReplayHandler.replayLog:236) - Replaying /flume2/data/log-19
> 25 Feb 2013 19:55:51,656 INFO [lifecycleSupervisor-1-0] (org.apache.flume.channel.file.LogFile$SequentialReader.skipToLastCheckpointPosition:466) - fast-forward to checkpoint position: 1622678590
> 25 Feb 2013 19:55:51,656 INFO [lifecycleSupervisor-1-0] (org.apache.flume.channel.file.ReplayHandler.replayLog:236) - Replaying /flume2/data/log-20
> 25 Feb 2013 19:55:51,657 INFO [lifecycleSupervisor-1-0] (org.apache.flume.channel.file.LogFile$SequentialReader.skipToLastCheckpointPosition:466) - fast-forward to checkpoint position: 244707334
> 25 Feb 2013 19:55:51,657 INFO [lifecycleSupervisor-1-0] (org.apache.flume.channel.file.ReplayHandler.replayLog:236) - Replaying /flume2/data/log-21
> 25 Feb 2013 19:55:51,657 INFO [lifecycleSupervisor-1-0] (org.apache.flume.channel.file.LogFile$SequentialReader.skipToLastCheckpointPosition:466) - fast-forward to checkpoint position: 530601497
> 25 Feb 2013 19:55:51,658 INFO [lifecycleSupervisor-1-0] (org.apache.flume.channel.file.LogFile$SequentialReader.next:491) - Encountered EOF at 530601497 in /flume2/data/log-21
> 25 Feb 2013 19:55:51,658 INFO [lifecycleSupervisor-1-0] (org.apache.flume.channel.file.ReplayHandler.replayLog:236) - Replaying /flume2/data/log-22
> 25 Feb 2013 19:55:51,658 INFO [lifecycleSupervisor-1-1] (org.apache.flume.channel.file.ReplayHandler.replayLog:236) - Replaying /flume1/data/log-18
> 25 Feb 2013 19:55:51,658 WARN [lifecycleSupervisor-1-0] (org.apache.flume.channel.file.LogFile$SequentialReader.skipToLastCheckpointPosition:470) - Checkpoint for file(/flume2/data/log-22) is: 1361844516782, which is beyond the requested checkpoint time: 1361844516783 and position 0
-
Re: File Channel error stops flume
Rahul Ravindran 2013-02-25, 21:20
Thanks Hari. I had waited for 20 minutes and this did not move change. Now, after more than an hour, I see it working
________________________________
From: Hari Shreedharan <[EMAIL PROTECTED]>
To: [EMAIL PROTECTED]; Rahul Ravindran <[EMAIL PROTECTED]>
Sent: Monday, February 25, 2013 12:46 PM
Subject: Re: File Channel error stops flume
Rahul,
Those messages actually just suggest that your channel is replaying. The channel will complete the replay and the agent will start the sinks once the channel is ready. It might take a few minutes based on how many events you have in the channel.
Hari
--
Hari Shreedharan
On Monday, February 25, 2013 at 12:07 PM, Rahul Ravindran wrote:
Hi,
> I modified a parameter to the HDFS sink on a flume config (added an idleInterval) on 2 machines. Things worked fine on one, and not on the other. I tried restarting flume a couple of times and I continue seeing the same log statement (bolded below) with no writes to HDFS
>
>
>25 Feb 2013 08:27:00,174 INFO [Log-BackgroundWorker-ch2] (org.apache.flume.channel.file.EventQueueBackingStoreFile.checkpoint:109) - Start checkpoint for /flume2/checkpoint/checkpoint, elements to sync = 8506
>:%
>25 Feb 2013 19:55:51,577 INFO [lifecycleSupervisor-1-0] (org.apache.flume.channel.file.ReplayHandler.replayLog:236) - Replaying /flume2/data/log-17
>25 Feb 2013 19:55:51,585 INFO [lifecycleSupervisor-1-1] (org.apache.flume.channel.file.ReplayHandler.replayLog:236) - Replaying /flume1/data/log-17
>25 Feb 2013 19:55:51,588 INFO [lifecycleSupervisor-1-0] (org.apache.flume.tools.DirectMemoryUtils.getDefaultDirectMemorySize:113) - Unable to get maxDirectMemory from VM: NoSuchMethodException: sun.misc.VM.maxDirectMemory(null)
>25 Feb 2013 19:55:51,592 INFO [lifecycleSupervisor-1-0] (org.apache.flume.tools.DirectMemoryUtils.allocate:47) - Direct Memory Allocation: Allocation = 1048576, Allocated = 0, MaxDirectMemorySize = 268435456, Remaining = 268435456
>25 Feb 2013 19:55:51,634 INFO [lifecycleSupervisor-1-1] (org.apache.flume.channel.file.LogFile$SequentialReader.skipToLastCheckpointPosition:466) - fast-forward to checkpoint position: 1622812128
>25 Feb 2013 19:55:51,634 INFO [lifecycleSupervisor-1-0] (org.apache.flume.channel.file.LogFile$SequentialReader.skipToLastCheckpointPosition:466) - fast-forward to checkpoint position: 1622720601
>25 Feb 2013 19:55:51,654 INFO [lifecycleSupervisor-1-0] (org.apache.flume.channel.file.ReplayHandler.replayLog:236) - Replaying /flume2/data/log-18
>25 Feb 2013 19:55:51,655 INFO [lifecycleSupervisor-1-0] (org.apache.flume.channel.file.LogFile$SequentialReader.skipToLastCheckpointPosition:466) - fast-forward to checkpoint position: 1622821593
>25 Feb 2013 19:55:51,655 INFO [lifecycleSupervisor-1-0] (org.apache.flume.channel.file.ReplayHandler.replayLog:236) - Replaying /flume2/data/log-19
>25 Feb 2013 19:55:51,656 INFO [lifecycleSupervisor-1-0] (org.apache.flume.channel.file.LogFile$SequentialReader.skipToLastCheckpointPosition:466) - fast-forward to checkpoint position: 1622678590
>25 Feb 2013 19:55:51,656 INFO [lifecycleSupervisor-1-0] (org.apache.flume.channel.file.ReplayHandler.replayLog:236) - Replaying /flume2/data/log-20
>25 Feb 2013 19:55:51,657 INFO [lifecycleSupervisor-1-0] (org.apache.flume.channel.file.LogFile$SequentialReader.skipToLastCheckpointPosition:466) - fast-forward to checkpoint position: 244707334
>25 Feb 2013 19:55:51,657 INFO [lifecycleSupervisor-1-0] (org.apache.flume.channel.file.ReplayHandler.replayLog:236) - Replaying /flume2/data/log-21
>25 Feb 2013 19:55:51,657 INFO [lifecycleSupervisor-1-0] (org.apache.flume.channel.file.LogFile$SequentialReader.skipToLastCheckpointPosition:466) - fast-forward to checkpoint position: 530601497
>25 Feb 2013 19:55:51,658 INFO [lifecycleSupervisor-1-0] (org.apache.flume.channel.file.LogFile$SequentialReader.next:491) - Encountered EOF at 530601497 in /flume2/data/log-21
>25 Feb 2013 19:55:51,658 INFO [lifecycleSupervisor-1-0] (org.apache.flume.channel.file.ReplayHandler.replayLog:236) - Replaying /flume2/data/log-22
-
Re: File Channel error stops flume
Hari Shreedharan 2013-02-25, 21:30
Can you send your full logs? I suspect the channel did a full replay because it was restarted during a restart. (If it did, the logs would show a BadCheckpointException).
Hari
--
Hari Shreedharan
On Monday, February 25, 2013 at 1:20 PM, Rahul Ravindran wrote:
> Thanks Hari. I had waited for 20 minutes and this did not move change. Now, after more than an hour, I see it working
>
> From: Hari Shreedharan <[EMAIL PROTECTED] (mailto:[EMAIL PROTECTED])>
> To: [EMAIL PROTECTED] (mailto:[EMAIL PROTECTED]); Rahul Ravindran <[EMAIL PROTECTED] (mailto:[EMAIL PROTECTED])>
> Sent: Monday, February 25, 2013 12:46 PM
> Subject: Re: File Channel error stops flume
>
> Rahul,
>
> Those messages actually just suggest that your channel is replaying. The channel will complete the replay and the agent will start the sinks once the channel is ready. It might take a few minutes based on how many events you have in the channel.
>
>
> Hari
>
> --
> Hari Shreedharan
>
> On Monday, February 25, 2013 at 12:07 PM, Rahul Ravindran wrote:
> > Hi,
> > I modified a parameter to the HDFS sink on a flume config (added an idleInterval) on 2 machines. Things worked fine on one, and not on the other. I tried restarting flume a couple of times and I continue seeing the same log statement (bolded below) with no writes to HDFS
> >
> > 25 Feb 2013 08:27:00,174 INFO [Log-BackgroundWorker-ch2] (org.apache.flume.channel.file.EventQueueBackingStoreFile.checkpoint:109) - Start checkpoint for /flume2/checkpoint/checkpoint, elements to sync = 8506
> > :%
> > 25 Feb 2013 19:55:51,577 INFO [lifecycleSupervisor-1-0] (org.apache.flume.channel.file.ReplayHandler.replayLog:236) - Replaying /flume2/data/log-17
> > 25 Feb 2013 19:55:51,585 INFO [lifecycleSupervisor-1-1] (org.apache.flume.channel.file.ReplayHandler.replayLog:236) - Replaying /flume1/data/log-17
> > 25 Feb 2013 19:55:51,588 INFO [lifecycleSupervisor-1-0] (org.apache.flume.tools.DirectMemoryUtils.getDefaultDirectMemorySize:113) - Unable to get maxDirectMemory from VM: NoSuchMethodException: sun.misc.VM.maxDirectMemory(null)
> > 25 Feb 2013 19:55:51,592 INFO [lifecycleSupervisor-1-0] (org.apache.flume.tools.DirectMemoryUtils.allocate:47) - Direct Memory Allocation: Allocation = 1048576, Allocated = 0, MaxDirectMemorySize = 268435456, Remaining = 268435456
> > 25 Feb 2013 19:55:51,634 INFO [lifecycleSupervisor-1-1] (org.apache.flume.channel.file.LogFile$SequentialReader.skipToLastCheckpointPosition:466) - fast-forward to checkpoint position: 1622812128
> > 25 Feb 2013 19:55:51,634 INFO [lifecycleSupervisor-1-0] (org.apache.flume.channel.file.LogFile$SequentialReader.skipToLastCheckpointPosition:466) - fast-forward to checkpoint position: 1622720601
> > 25 Feb 2013 19:55:51,654 INFO [lifecycleSupervisor-1-0] (org.apache.flume.channel.file.ReplayHandler.replayLog:236) - Replaying /flume2/data/log-18
> > 25 Feb 2013 19:55:51,655 INFO [lifecycleSupervisor-1-0] (org.apache.flume.channel.file.LogFile$SequentialReader.skipToLastCheckpointPosition:466) - fast-forward to checkpoint position: 1622821593
> > 25 Feb 2013 19:55:51,655 INFO [lifecycleSupervisor-1-0] (org.apache.flume.channel.file.ReplayHandler.replayLog:236) - Replaying /flume2/data/log-19
> > 25 Feb 2013 19:55:51,656 INFO [lifecycleSupervisor-1-0] (org.apache.flume.channel.file.LogFile$SequentialReader.skipToLastCheckpointPosition:466) - fast-forward to checkpoint position: 1622678590
> > 25 Feb 2013 19:55:51,656 INFO [lifecycleSupervisor-1-0] (org.apache.flume.channel.file.ReplayHandler.replayLog:236) - Replaying /flume2/data/log-20
> > 25 Feb 2013 19:55:51,657 INFO [lifecycleSupervisor-1-0] (org.apache.flume.channel.file.LogFile$SequentialReader.skipToLastCheckpointPosition:466) - fast-forward to checkpoint position: 244707334
> > 25 Feb 2013 19:55:51,657 INFO [lifecycleSupervisor-1-0] (org.apache.flume.channel.file.ReplayHandler.replayLog:236) - Replaying /flume2/data/log-21
> > 25 Feb 2013 19:55:51,657 INFO [lifecycleSupervisor-1-0] (org.apache.flume.channel.file.LogFile$SequentialReader.skipToLastCheckpointPosition:466) - fast-forward to checkpoint position: 530601497
-
Re: File Channel error stops flume
Rahul Ravindran 2013-02-26, 00:28
I have attached the zipped log file at https://issues.apache.org/jira/browse/FLUME-1928________________________________ From: Hari Shreedharan <[EMAIL PROTECTED]> To: [EMAIL PROTECTED]; Rahul Ravindran <[EMAIL PROTECTED]> Sent: Monday, February 25, 2013 1:30 PM Subject: Re: File Channel error stops flume Can you send your full logs? I suspect the channel did a full replay because it was restarted during a restart. (If it did, the logs would show a BadCheckpointException). Hari -- Hari Shreedharan On Monday, February 25, 2013 at 1:20 PM, Rahul Ravindran wrote: Thanks Hari. I had waited for 20 minutes and this did not move change. Now, after more than an hour, I see it working > > > >________________________________ > From: Hari Shreedharan <[EMAIL PROTECTED]> >To: [EMAIL PROTECTED]; Rahul Ravindran <[EMAIL PROTECTED]> >Sent: Monday, February 25, 2013 12:46 PM >Subject: Re: File Channel error stops flume > > >Rahul, > > >Those messages actually just suggest that your channel is replaying. The channel will complete the replay and the agent will start the sinks once the channel is ready. It might take a few minutes based on how many events you have in the channel. > > > > >Hari > > >-- >Hari Shreedharan > > >On Monday, February 25, 2013 at 12:07 PM, Rahul Ravindran wrote: >Hi, >> I modified a parameter to the HDFS sink on a flume config (added an idleInterval) on 2 machines. Things worked fine on one, and not on the other. I tried restarting flume a couple of times and I continue seeing the same log statement (bolded below) with no writes to HDFS >> >> >>25 Feb 2013 08:27:00,174 INFO [Log-BackgroundWorker-ch2] (org.apache.flume.channel.file.EventQueueBackingStoreFile.checkpoint:109) - Start checkpoint for /flume2/checkpoint/checkpoint, elements to sync = 8506 >>:% >>25 Feb 2013 19:55:51,577 INFO [lifecycleSupervisor-1-0] (org.apache.flume.channel.file.ReplayHandler.replayLog:236) - Replaying /flume2/data/log-17 >>25 Feb 2013 19:55:51,585 INFO [lifecycleSupervisor-1-1] (org.apache.flume.channel.file.ReplayHandler.replayLog:236) - Replaying /flume1/data/log-17 >>25 Feb 2013 19:55:51,588 INFO [lifecycleSupervisor-1-0] (org.apache.flume.tools.DirectMemoryUtils.getDefaultDirectMemorySize:113) - Unable to get maxDirectMemory from VM: NoSuchMethodException: sun.misc.VM.maxDirectMemory(null) >>25 Feb 2013 19:55:51,592 INFO [lifecycleSupervisor-1-0] (org.apache.flume.tools.DirectMemoryUtils.allocate:47) - Direct Memory Allocation: Allocation = 1048576, Allocated = 0, MaxDirectMemorySize = 268435456, Remaining = 268435456 >>25 Feb 2013 19:55:51,634 INFO [lifecycleSupervisor-1-1] (org.apache.flume.channel.file.LogFile$SequentialReader.skipToLastCheckpointPosition:466) - fast-forward to checkpoint position: 1622812128 >>25 Feb 2013 19:55:51,634 INFO [lifecycleSupervisor-1-0] (org.apache.flume.channel.file.LogFile$SequentialReader.skipToLastCheckpointPosition:466) - fast-forward to checkpoint position: 1622720601 >>25 Feb 2013 19:55:51,654 INFO [lifecycleSupervisor-1-0] (org.apache.flume.channel.file.ReplayHandler.replayLog:236) - Replaying /flume2/data/log-18 >>25 Feb 2013 19:55:51,655 INFO [lifecycleSupervisor-1-0] (org.apache.flume.channel.file.LogFile$SequentialReader.skipToLastCheckpointPosition:466) - fast-forward to checkpoint position: 1622821593 >>25 Feb 2013 19:55:51,655 INFO [lifecycleSupervisor-1-0] (org.apache.flume.channel.file.ReplayHandler.replayLog:236) - Replaying /flume2/data/log-19 >>25 Feb 2013 19:55:51,656 INFO [lifecycleSupervisor-1-0] (org.apache.flume.channel.file.LogFile$SequentialReader.skipToLastCheckpointPosition:466) - fast-forward to checkpoint position: 1622678590 >>25 Feb 2013 19:55:51,656 INFO [lifecycleSupervisor-1-0] (org.apache.flume.channel.file.ReplayHandler.replayLog:236) - Replaying /flume2/data/log-20 >>25 Feb 2013 19:55:51,657 INFO [lifecycleSupervisor-1-0] (org.apache.flume.channel.file.LogFile$SequentialReader.skipToLastCheckpointPosition:466) - fast-forward to checkpoint position: 244707334
-
Re: File Channel error stops flume
Hari Shreedharan 2013-03-01, 22:57
>From the logs, the logs look OK and the channel is working fine. It seems to have been replaying - that is pretty much it. -- Hari Shreedharan On Monday, February 25, 2013 at 4:28 PM, Rahul Ravindran wrote: > I have attached the zipped log file at > https://issues.apache.org/jira/browse/FLUME-1928> > From: Hari Shreedharan <[EMAIL PROTECTED] (mailto:[EMAIL PROTECTED])> > To: [EMAIL PROTECTED] (mailto:[EMAIL PROTECTED]); Rahul Ravindran <[EMAIL PROTECTED] (mailto:[EMAIL PROTECTED])> > Sent: Monday, February 25, 2013 1:30 PM > Subject: Re: File Channel error stops flume > > Can you send your full logs? I suspect the channel did a full replay because it was restarted during a restart. (If it did, the logs would show a BadCheckpointException). > > > Hari > > -- > Hari Shreedharan > > On Monday, February 25, 2013 at 1:20 PM, Rahul Ravindran wrote: > > Thanks Hari. I had waited for 20 minutes and this did not move change. Now, after more than an hour, I see it working > > > > From: Hari Shreedharan <[EMAIL PROTECTED] (mailto:[EMAIL PROTECTED])> > > To: [EMAIL PROTECTED] (mailto:[EMAIL PROTECTED]); Rahul Ravindran <[EMAIL PROTECTED] (mailto:[EMAIL PROTECTED])> > > Sent: Monday, February 25, 2013 12:46 PM > > Subject: Re: File Channel error stops flume > > > > Rahul, > > > > Those messages actually just suggest that your channel is replaying. The channel will complete the replay and the agent will start the sinks once the channel is ready. It might take a few minutes based on how many events you have in the channel. > > > > > > Hari > > > > -- > > Hari Shreedharan > > > > On Monday, February 25, 2013 at 12:07 PM, Rahul Ravindran wrote: > > > Hi, > > > I modified a parameter to the HDFS sink on a flume config (added an idleInterval) on 2 machines. Things worked fine on one, and not on the other. I tried restarting flume a couple of times and I continue seeing the same log statement (bolded below) with no writes to HDFS > > > > > > 25 Feb 2013 08:27:00,174 INFO [Log-BackgroundWorker-ch2] (org.apache.flume.channel.file.EventQueueBackingStoreFile.checkpoint:109) - Start checkpoint for /flume2/checkpoint/checkpoint, elements to sync = 8506 > > > :% > > > 25 Feb 2013 19:55:51,577 INFO [lifecycleSupervisor-1-0] (org.apache.flume.channel.file.ReplayHandler.replayLog:236) - Replaying /flume2/data/log-17 > > > 25 Feb 2013 19:55:51,585 INFO [lifecycleSupervisor-1-1] (org.apache.flume.channel.file.ReplayHandler.replayLog:236) - Replaying /flume1/data/log-17 > > > 25 Feb 2013 19:55:51,588 INFO [lifecycleSupervisor-1-0] (org.apache.flume.tools.DirectMemoryUtils.getDefaultDirectMemorySize:113) - Unable to get maxDirectMemory from VM: NoSuchMethodException: sun.misc.VM.maxDirectMemory(null) > > > 25 Feb 2013 19:55:51,592 INFO [lifecycleSupervisor-1-0] (org.apache.flume.tools.DirectMemoryUtils.allocate:47) - Direct Memory Allocation: Allocation = 1048576, Allocated = 0, MaxDirectMemorySize = 268435456, Remaining = 268435456 > > > 25 Feb 2013 19:55:51,634 INFO [lifecycleSupervisor-1-1] (org.apache.flume.channel.file.LogFile$SequentialReader.skipToLastCheckpointPosition:466) - fast-forward to checkpoint position: 1622812128 > > > 25 Feb 2013 19:55:51,634 INFO [lifecycleSupervisor-1-0] (org.apache.flume.channel.file.LogFile$SequentialReader.skipToLastCheckpointPosition:466) - fast-forward to checkpoint position: 1622720601 > > > 25 Feb 2013 19:55:51,654 INFO [lifecycleSupervisor-1-0] (org.apache.flume.channel.file.ReplayHandler.replayLog:236) - Replaying /flume2/data/log-18 > > > 25 Feb 2013 19:55:51,655 INFO [lifecycleSupervisor-1-0] (org.apache.flume.channel.file.LogFile$SequentialReader.skipToLastCheckpointPosition:466) - fast-forward to checkpoint position: 1622821593 > > > 25 Feb 2013 19:55:51,655 INFO [lifecycleSupervisor-1-0] (org.apache.flume.channel.file.ReplayHandler.replayLog:236) - Replaying /flume2/data/log-19 > > > 25 Feb 2013 19:55:51,656 INFO [lifecycleSupervisor-1-0] (org.apache.flume.channel.file.LogFile$SequentialReader.skipToLastCheckpointPosition:466) - fast-forward to checkpoint position: 1622678590
|
|
