Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Flume, mail # user - Usage of  use-fast-replay for FileChannel


+
Rahul Ravindran 2013-05-07, 04:40
+
Hari Shreedharan 2013-05-07, 04:57
Copy link to this message
-
Re: Usage of use-fast-replay for FileChannel
Rahul Ravindran 2013-05-07, 15:24
I searched through the logs but did not find a BadCheckpointException. I extrapolated the 6G from the size of the log-* files being read. Below is a snippet of the logs. As you can see from the timestamp of the log lines, the channel ch3 was started at time 3:32 and channel startup activity completed only at 4:40

07 May 2013 03:32:40,598 INFO  [lifecycleSupervisor-1-1] (org.apache.flume.channel.file.LogFileV3$MetaDataWriter.markCheckpoint:85)  - Updating log-297.meta currentPosition = 1455187648, logWriteOrderID = 1385582495370
07 May 2013 03:32:40,599 INFO  [lifecycleSupervisor-1-1] (org.apache.flume.channel.file.FileChannel.start:309)  - Queue Size after replay: 58497343 [channel=ch3]
07 May 2013 03:32:40,628 INFO  [lifecycleSupervisor-1-1] (org.apache.flume.instrumentation.MonitoredCounterGroup.register:89)  - Monitoried counter group for type: CHANNEL, name: ch3, registered successfully.
07 May 2013 03:32:40,628 INFO  [lifecycleSupervisor-1-1] (org.apache.flume.instrumentation.MonitoredCounterGroup.start:73)  - Component type: CHANNEL, name: ch3 started
07 May 2013 03:32:40,887 INFO  [lifecycleSupervisor-1-3] (org.apache.flume.channel.file.LogFile$SequentialReader.next:491)  - Encountered EOF at 1623195640 in /flume2/data/log-861
07 May 2013 03:32:45,972 INFO  [lifecycleSupervisor-1-3] (org.apache.flume.channel.file.LogFile$SequentialReader.next:491)  - Encountered EOF at 1623195632 in /flume2/data/log-862
07 May 2013 03:32:47,777 INFO  [lifecycleSupervisor-1-3] (org.apache.flume.channel.file.LogFile$SequentialReader.next:491)  - Encountered EOF at 1623195646 in /flume2/data/log-863
07 May 2013 03:32:49,349 INFO  [lifecycleSupervisor-1-3] (org.apache.flume.channel.file.LogFile$SequentialReader.next:491)  - Encountered EOF at 1623195620 in /flume2/data/log-864
07 May 2013 04:17:37,858 INFO  [lifecycleSupervisor-1-3] (org.apache.flume.channel.file.LogFile$SequentialReader.next:491)  - Encountered EOF at 1442343132 in /flume2/data/log-865
07 May 2013 04:17:37,859 INFO  [lifecycleSupervisor-1-3] (org.apache.flume.channel.file.ReplayHandler.replayLog:320)  - read: 1829658, put: 0, take: 45223, rollback: 1, commit: 11, skip: 1784423, eventCount:44000
07 May 2013 04:17:37,860 ERROR [lifecycleSupervisor-1-3] (org.apache.flume.channel.file.ReplayHandler.replayLog:354)  - Pending takes 1473 exist after the end of replay. Duplicate messages will exist in destination.
07 May 2013 04:17:37,860 INFO  [lifecycleSupervisor-1-3] (org.apache.flume.channel.file.Log.replay:399)  - Rolling /flume2/data
07 May 2013 04:17:37,860 INFO  [lifecycleSupervisor-1-3] (org.apache.flume.channel.file.Log.roll:811)  - Roll start /flume2/data
07 May 2013 04:17:37,861 INFO  [lifecycleSupervisor-1-3] (org.apache.flume.channel.file.LogFile$Writer.<init>:171)  - Opened /flume2/data/log-866
07 May 2013 04:17:37,862 INFO  [lifecycleSupervisor-1-3] (org.apache.flume.channel.file.Log.roll:826)  - Roll end
07 May 2013 04:17:37,863 INFO  [lifecycleSupervisor-1-3] (org.apache.flume.channel.file.EventQueueBackingStoreFile.checkpoint:109)  - Start checkpoint for /flume2/checkpoint/checkpoint, elements to sync = 42527
07 May 2013 04:17:37,865 INFO  [lifecycleSupervisor-1-3] (org.apache.flume.channel.file.EventQueueBackingStoreFile.checkpoint:117)  - Updating checkpoint metadata: logWriteOrderID: 1385582495371, queueSize: 46200121, queueHead: 65890565
07 May 2013 04:17:37,906 INFO  [lifecycleSupervisor-1-3] (org.apache.flume.channel.file.LogFileV3$MetaDataWriter.markCheckpoint:85)  - Updating log-866.meta currentPosition = 0, logWriteOrderID = 1385582495371
07 May 2013 04:17:37,907 INFO  [lifecycleSupervisor-1-3] (org.apache.flume.channel.file.Log.writeCheckpoint:886)  - Updated checkpoint for file: /flume2/data/log-866 position: 0 logWriteOrderID: 1385582495371
07 May 2013 04:17:37,908 INFO  [lifecycleSupervisor-1-3] (org.apache.flume.channel.file.LogFile$RandomReader.close:356)  - Closing RandomReader /flume2/data/log-855
07 May 2013 04:17:37,913 INFO  [lifecycleSupervisor-1-3] (org.apache.flume.channel.file.LogFileV3$MetaDataWriter.markCheckpoint:85)  - Updating log-855.meta currentPosition = 1592727635, logWriteOrderID = 1385582495371
07 May 2013 04:17:37,915 INFO  [lifecycleSupervisor-1-3] (org.apache.flume.channel.file.LogFile$RandomReader.close:356)  - Closing RandomReader /flume2/data/log-856
07 May 2013 04:17:37,921 INFO  [lifecycleSupervisor-1-3] (org.apache.flume.channel.file.LogFileV3$MetaDataWriter.markCheckpoint:85)  - Updating log-856.meta currentPosition = 1601406592, logWriteOrderID = 1385582495371
07 May 2013 04:17:37,922 INFO  [lifecycleSupervisor-1-3] (org.apache.flume.channel.file.LogFile$RandomReader.close:356)  - Closing RandomReader /flume2/data/log-857
07 May 2013 04:17:37,928 INFO  [lifecycleSupervisor-1-3] (org.apache.flume.channel.file.LogFileV3$MetaDataWriter.markCheckpoint:85)  - Updating log-857.meta currentPosition = 1608758564, logWriteOrderID = 1385582495371
07 May 2013 04:17:37,929 INFO  [lifecycleSupervisor-1-3] (org.apache.flume.channel.file.LogFile$RandomReader.close:356)  - Closing RandomReader /flume2/data/log-858
07 May 2013 04:17:37,935 INFO  [lifecycleSupervisor-1-3] (org.apache.flume.channel.file.LogFileV3$MetaDataWriter.markCheckpoint:85)  - Updating log-858.meta currentPosition = 1558785242, logWriteOrderID = 1385582495371
07 May 2013 04:17:37,936 INFO  [lifecycleSupervisor-1-3] (org.apache.flume.channel.file.LogFile$RandomReader.close:356)  - Closing RandomReader /flume2/data/log-859
07 May 2013 04:17:37,941 INFO  [lifecycleSupervisor-1-3] (org.apache.flume.channel.file.LogFileV3$MetaDataWriter.markCheckpoint:85)  - Updating log-859.meta currentPosition = 586542373, logWriteOrderID = 1385582495371
07 May 2013 04:17:37,943 INFO  [lifecycleSupervisor-1-3] (org.apache.flume.channel.file.LogFile$RandomReader.close:356)  - Closing RandomReader /flume2/data/log-861
07 May 2013 04:17:37,949 INFO  [lifecycleSupervisor-1-3] (org.apache.flume.channel.file.LogFileV3$MetaDataWriter.markCheckpoint: