Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Flume, mail # dev - Review Request: FLUME-1516. Write dual checkpoints.


+
Hari Shreedharan 2013-01-09, 07:06
+
Hari Shreedharan 2013-01-09, 17:09
+
Hari Shreedharan 2013-01-21, 21:36
+
Hari Shreedharan 2013-01-22, 00:43
+
Hari Shreedharan 2013-03-04, 20:44
+
Hari Shreedharan 2013-03-14, 07:31
+
Brock Noland 2013-03-14, 17:18
+
Hari Shreedharan 2013-03-14, 18:10
+
Brock Noland 2013-03-14, 18:37
+
Hari Shreedharan 2013-03-14, 23:44
Copy link to this message
-
Re: Review Request: FLUME-1516. Write dual checkpoints.
Brock Noland 2013-03-21, 03:01

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/8899/#review18196
-----------------------------------------------------------
Looks great Hari!

One item I am not convinced we need is the backupCheckpoint{writeorderid/position}. It adds complexity for not a terrible amount of benefit.  In the case we cannot seek into the file we will have to read all the logs. However, we won't have to process the logs in terms of the queue which is the big bottleneck. What do you think about removing those fields to make this simpler?
flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/EventQueueBackingStore.java
<https://reviews.apache.org/r/8899/#comment38385>

    The changes to this class seem like they belong in EventQueueBackingStoreFile.

flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/EventQueueBackingStoreFile.java
<https://reviews.apache.org/r/8899/#comment38383>

    This file create could fail. We should log if it does.

flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/EventQueueBackingStoreFile.java
<https://reviews.apache.org/r/8899/#comment38381>

    Since this a Throwable and not an Exception I think the variable should be throwable as opposed to ex.

flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FileChannel.java
<https://reviews.apache.org/r/8899/#comment38384>

    As opposed to allowing this to be null and complicating the check below, how about passing in "" as a default and calling trim() on the returned value?

flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Log.java
<https://reviews.apache.org/r/8899/#comment38386>

    I think "filesToDelete" would be better named "pendingDeletes" or something like that.

flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Log.java
<https://reviews.apache.org/r/8899/#comment38387>

    Why not have a single code path? It will change the current behavior slightly but I don't see a huge issue with that.
- Brock Noland
On March 14, 2013, 11:43 p.m., Hari Shreedharan wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/8899/
> -----------------------------------------------------------
>
> (Updated March 14, 2013, 11:43 p.m.)
>
>
> Review request for Flume.
>
>
> Description
> -------
>
> Added support for backup and retrieval of checkpoint.
>
>
> This addresses bug FLUME-1516.
>     https://issues.apache.org/jira/browse/FLUME-1516
>
>
> Diffs
> -----
>
>   flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/EventQueueBackingStore.java b136eb0
>   flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/EventQueueBackingStoreFactory.java a19bdb4
>   flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/EventQueueBackingStoreFile.java 4115505
>   flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/EventQueueBackingStoreFileV3.java 451a9d4
>   flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FileChannel.java ff42d19
>   flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FileChannelConfiguration.java 24368b3
>   flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FlumeEventQueue.java 1ed9547
>   flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Log.java 6ffc824
>   flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/LogFile.java 1db3717
>   flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/LogFileV3.java f51935c
>   flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/ReplayHandler.java fa4fd9d
+
Hari Shreedharan 2013-03-21, 21:30
+
Hari Shreedharan 2013-03-25, 21:03
+
Brock Noland 2013-03-25, 21:51
+
Hari Shreedharan 2013-03-25, 22:08
+
Brock Noland 2013-03-25, 22:12
+
Hari Shreedharan 2013-03-26, 22:52
+
Brock Noland 2013-04-04, 18:20
+
Hari Shreedharan 2013-04-04, 21:34
+
Hari Shreedharan 2013-04-04, 21:32
+
Hari Shreedharan 2013-04-05, 00:16
+
Brock Noland 2013-04-05, 14:35
+
Hari Shreedharan 2013-04-05, 18:01
+
Brock Noland 2013-04-05, 18:26