Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Flume >> mail # user >> Usage of  use-fast-replay for FileChannel


Copy link to this message
-
Re: Usage of use-fast-replay for FileChannel
I searched through the logs but did not find a BadCheckpointException. I extrapolated the 6G from the size of the log-* files being read. Below is a snippet of the logs. As you can see from the timestamp of the log lines, the channel ch3 was started at time 3:32 and channel startup activity completed only at 4:40

07 May 2013 03:32:40,598 INFO  [lifecycleSupervisor-1-1] (org.apache.flume.channel.file.LogFileV3$MetaDataWriter.markCheckpoint:85)  - Updating log-297.meta currentPosition = 1455187648, logWriteOrderID = 1385582495370
07 May 2013 03:32:40,599 INFO  [lifecycleSupervisor-1-1] (org.apache.flume.channel.file.FileChannel.start:309)  - Queue Size after replay: 58497343 [channel=ch3]
07 May 2013 03:32:40,628 INFO  [lifecycleSupervisor-1-1] (org.apache.flume.instrumentation.MonitoredCounterGroup.register:89)  - Monitoried counter group for type: CHANNEL, name: ch3, registered successfully.
07 May 2013 03:32:40,628 INFO  [lifecycleSupervisor-1-1] (org.apache.flume.instrumentation.MonitoredCounterGroup.start:73)  - Component type: CHANNEL, name: ch3 started
07 May 2013 03:32:40,887 INFO  [lifecycleSupervisor-1-3] (org.apache.flume.channel.file.LogFile$SequentialReader.next:491)  - Encountered EOF at 1623195640 in /flume2/data/log-861
07 May 2013 03:32:45,972 INFO  [lifecycleSupervisor-1-3] (org.apache.flume.channel.file.LogFile$SequentialReader.next:491)  - Encountered EOF at 1623195632 in /flume2/data/log-862
07 May 2013 03:32:47,777 INFO  [lifecycleSupervisor-1-3] (org.apache.flume.channel.file.LogFile$SequentialReader.next:491)  - Encountered EOF at 1623195646 in /flume2/data/log-863
07 May 2013 03:32:49,349 INFO  [lifecycleSupervisor-1-3] (org.apache.flume.channel.file.LogFile$SequentialReader.next:491)  - Encountered EOF at 1623195620 in /flume2/data/log-864
07 May 2013 04:17:37,858 INFO  [lifecycleSupervisor-1-3] (org.apache.flume.channel.file.LogFile$SequentialReader.next:491)  - Encountered EOF at 1442343132 in /flume2/data/log-865
07 May 2013 04:17:37,859 INFO  [lifecycleSupervisor-1-3] (org.apache.flume.channel.file.ReplayHandler.replayLog:320)  - read: 1829658, put: 0, take: 45223, rollback: 1, commit: 11, skip: 1784423, eventCount:44000
07 May 2013 04:17:37,860 ERROR [lifecycleSupervisor-1-3] (org.apache.flume.channel.file.ReplayHandler.replayLog:354)  - Pending takes 1473 exist after the end of replay. Duplicate messages will exist in destination.
07 May 2013 04:17:37,860 INFO  [lifecycleSupervisor-1-3] (org.apache.flume.channel.file.Log.replay:399)  - Rolling /flume2/data
07 May 2013 04:17:37,860 INFO  [lifecycleSupervisor-1-3] (org.apache.flume.channel.file.Log.roll:811)  - Roll start /flume2/data
07 May 2013 04:17:37,861 INFO  [lifecycleSupervisor-1-3] (org.apache.flume.channel.file.LogFile$Writer.<init>:171)  - Opened /flume2/data/log-866
07 May 2013 04:17:37,862 INFO  [lifecycleSupervisor-1-3] (org.apache.flume.channel.file.Log.roll:826)  - Roll end
07 May 2013 04:17:37,863 INFO  [lifecycleSupervisor-1-3] (org.apache.flume.channel.file.EventQueueBackingStoreFile.checkpoint:109)  - Start checkpoint for /flume2/checkpoint/checkpoint, elements to sync = 42527
07 May 2013 04:17:37,865 INFO  [lifecycleSupervisor-1-3] (org.apache.flume.channel.file.EventQueueBackingStoreFile.checkpoint:117)  - Updating checkpoint metadata: logWriteOrderID: 1385582495371, queueSize: 46200121, queueHead: 65890565
07 May 2013 04:17:37,906 INFO  [lifecycleSupervisor-1-3] (org.apache.flume.channel.file.LogFileV3$MetaDataWriter.markCheckpoint:85)  - Updating log-866.meta currentPosition = 0, logWriteOrderID = 1385582495371
07 May 2013 04:17:37,907 INFO  [lifecycleSupervisor-1-3] (org.apache.flume.channel.file.Log.writeCheckpoint:886)  - Updated checkpoint for file: /flume2/data/log-866 position: 0 logWriteOrderID: 1385582495371
07 May 2013 04:17:37,908 INFO  [lifecycleSupervisor-1-3] (org.apache.flume.channel.file.LogFile$RandomReader.close:356)  - Closing RandomReader /flume2/data/log-855
07 May 2013 04:17:37,913 INFO  [lifecycleSupervisor-1-3] (org.apache.flume.channel.file.LogFileV3$MetaDataWriter.markCheckpoint:85)  - Updating log-855.meta currentPosition = 1592727635, logWriteOrderID = 1385582495371
07 May 2013 04:17:37,915 INFO  [lifecycleSupervisor-1-3] (org.apache.flume.channel.file.LogFile$RandomReader.close:356)  - Closing RandomReader /flume2/data/log-856
07 May 2013 04:17:37,921 INFO  [lifecycleSupervisor-1-3] (org.apache.flume.channel.file.LogFileV3$MetaDataWriter.markCheckpoint:85)  - Updating log-856.meta currentPosition = 1601406592, logWriteOrderID = 1385582495371
07 May 2013 04:17:37,922 INFO  [lifecycleSupervisor-1-3] (org.apache.flume.channel.file.LogFile$RandomReader.close:356)  - Closing RandomReader /flume2/data/log-857
07 May 2013 04:17:37,928 INFO  [lifecycleSupervisor-1-3] (org.apache.flume.channel.file.LogFileV3$MetaDataWriter.markCheckpoint:85)  - Updating log-857.meta currentPosition = 1608758564, logWriteOrderID = 1385582495371
07 May 2013 04:17:37,929 INFO  [lifecycleSupervisor-1-3] (org.apache.flume.channel.file.LogFile$RandomReader.close:356)  - Closing RandomReader /flume2/data/log-858
07 May 2013 04:17:37,935 INFO  [lifecycleSupervisor-1-3] (org.apache.flume.channel.file.LogFileV3$MetaDataWriter.markCheckpoint:85)  - Updating log-858.meta currentPosition = 1558785242, logWriteOrderID = 1385582495371
07 May 2013 04:17:37,936 INFO  [lifecycleSupervisor-1-3] (org.apache.flume.channel.file.LogFile$RandomReader.close:356)  - Closing RandomReader /flume2/data/log-859
07 May 2013 04:17:37,941 INFO  [lifecycleSupervisor-1-3] (org.apache.flume.channel.file.LogFileV3$MetaDataWriter.markCheckpoint:85)  - Updating log-859.meta currentPosition = 586542373, logWriteOrderID = 1385582495371
07 May 2013 04:17:37,943 INFO  [lifecycleSupervisor-1-3] (org.apache.flume.channel.file.LogFile$RandomReader.close:356)  - Closing RandomReader /flume2/data/log-861
07 May 2013 04:17:37,949 INFO  [lifecycleSupervisor-1-3] (org.apache.flume.channel.file.LogFileV3$MetaDataWriter.markCheckpoint:
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB