Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Flume, mail # dev - Review Request: FLUME-1516. Write dual checkpoints.


+
Hari Shreedharan 2013-01-09, 07:06
+
Hari Shreedharan 2013-01-09, 17:09
+
Hari Shreedharan 2013-01-21, 21:36
+
Hari Shreedharan 2013-01-22, 00:43
+
Hari Shreedharan 2013-03-04, 20:44
+
Hari Shreedharan 2013-03-14, 07:31
+
Brock Noland 2013-03-14, 17:18
+
Hari Shreedharan 2013-03-14, 18:10
Copy link to this message
-
Re: Review Request: FLUME-1516. Write dual checkpoints.
Brock Noland 2013-03-14, 18:37


> On March 14, 2013, 6:10 p.m., Hari Shreedharan wrote:
> > flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/EventQueueBackingStoreFile.java, line 154
> > <https://reviews.apache.org/r/8899/diff/6/?file=270479#file270479line154>
> >
> >     The reason there is a backupComplete file is to ensure that we know that all files were copied. The idea is that if we were in the middle of copying (only some files were copied, not all), then this indicates the copying was incomplete. This would be for the case where the checkpoint is somehow corrupt and we try to retrieve the backup - at that point, we need to be sure the backup actually contains all files required (this makes it extensible, in case we add more files to the checkpoint)

That is fine, I just meant the same file object exists a few lines up and we should handle the case where the create fails.
> On March 14, 2013, 6:10 p.m., Hari Shreedharan wrote:
> > flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/EventQueueBackingStoreFile.java, line 185
> > <https://reviews.apache.org/r/8899/diff/6/?file=270479#file270479line185>
> >
> >     The previous backup is still happening, so we skip this checkpoint, rather than trying to abort the backup, cleanup the previous one and start the checkpoint. (If the checkpoint interval is too low, then there may never be a backup if we do that)

Ok great, then we should clarify this in the msg.
- Brock
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/8899/#review17888
-----------------------------------------------------------
On March 14, 2013, 7:31 a.m., Hari Shreedharan wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/8899/
> -----------------------------------------------------------
>
> (Updated March 14, 2013, 7:31 a.m.)
>
>
> Review request for Flume.
>
>
> Description
> -------
>
> Added support for backup and retrieval of checkpoint.
>
>
> This addresses bug FLUME-1516.
>     https://issues.apache.org/jira/browse/FLUME-1516
>
>
> Diffs
> -----
>
>   flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/EventQueueBackingStore.java b136eb0
>   flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/EventQueueBackingStoreFactory.java a19bdb4
>   flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/EventQueueBackingStoreFile.java 4115505
>   flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/EventQueueBackingStoreFileV3.java 451a9d4
>   flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FileChannel.java ff42d19
>   flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FileChannelConfiguration.java 24368b3
>   flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FlumeEventQueue.java 1ed9547
>   flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Log.java 6ffc824
>   flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/LogFile.java 1db3717
>   flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/LogFileV3.java f51935c
>   flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/ReplayHandler.java fa4fd9d
>   flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Serialization.java 7094d3c
>   flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/proto/ProtosFactory.java e6d4957
>   flume-ng-channels/flume-file-channel/src/main/proto/filechannel.proto 3a4e828
>   flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFileChannelBase.java 3da09ab
>   flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFileChannelRestart.java 170dc72
+
Hari Shreedharan 2013-03-14, 23:44
+
Brock Noland 2013-03-21, 03:01
+
Hari Shreedharan 2013-03-21, 21:30
+
Hari Shreedharan 2013-03-25, 21:03
+
Brock Noland 2013-03-25, 21:51
+
Hari Shreedharan 2013-03-25, 22:08
+
Brock Noland 2013-03-25, 22:12
+
Hari Shreedharan 2013-03-26, 22:52
+
Brock Noland 2013-04-04, 18:20
+
Hari Shreedharan 2013-04-04, 21:34
+
Hari Shreedharan 2013-04-04, 21:32
+
Hari Shreedharan 2013-04-05, 00:16
+
Brock Noland 2013-04-05, 14:35
+
Hari Shreedharan 2013-04-05, 18:01
+
Brock Noland 2013-04-05, 18:26