Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Flume, mail # dev - Review Request: FLUME-1516. Write dual checkpoints.


+
Hari Shreedharan 2013-01-09, 07:06
+
Hari Shreedharan 2013-01-09, 17:09
+
Hari Shreedharan 2013-01-21, 21:36
+
Hari Shreedharan 2013-01-22, 00:43
+
Hari Shreedharan 2013-03-04, 20:44
+
Hari Shreedharan 2013-03-14, 07:31
+
Brock Noland 2013-03-14, 17:18
+
Hari Shreedharan 2013-03-14, 18:10
+
Brock Noland 2013-03-14, 18:37
+
Hari Shreedharan 2013-03-14, 23:44
+
Brock Noland 2013-03-21, 03:01
Copy link to this message
-
Re: Review Request: FLUME-1516. Write dual checkpoints.
Hari Shreedharan 2013-03-21, 21:30


> On March 21, 2013, 3:01 a.m., Brock Noland wrote:
> > Looks great Hari!
> >
> > One item I am not convinced we need is the backupCheckpoint{writeorderid/position}. It adds complexity for not a terrible amount of benefit.  In the case we cannot seek into the file we will have to read all the logs. However, we won't have to process the logs in terms of the queue which is the big bottleneck. What do you think about removing those fields to make this simpler?

I'd like to keep those, since we can avoid seeks. I have seen situations where there have been like 400 files, and just reading the entire files will still take a very long time. The changes related to the seek are still relatively small and there is a unit test to keep track of this. If we don't keep those, we will read all files fully - I think we should avoid that.
> On March 21, 2013, 3:01 a.m., Brock Noland wrote:
> > flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/EventQueueBackingStore.java, line 49
> > <https://reviews.apache.org/r/8899/diff/7/?file=270820#file270820line49>
> >
> >     The changes to this class seem like they belong in EventQueueBackingStoreFile.

Yep. Will move it.
> On March 21, 2013, 3:01 a.m., Brock Noland wrote:
> > flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/EventQueueBackingStoreFile.java, line 159
> > <https://reviews.apache.org/r/8899/diff/7/?file=270822#file270822line159>
> >
> >     This file create could fail. We should log if it does.

Sure. Will add this
> On March 21, 2013, 3:01 a.m., Brock Noland wrote:
> > flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/EventQueueBackingStoreFile.java, line 253
> > <https://reviews.apache.org/r/8899/diff/7/?file=270822#file270822line253>
> >
> >     Since this a Throwable and not an Exception I think the variable should be throwable as opposed to ex.

Will do.
> On March 21, 2013, 3:01 a.m., Brock Noland wrote:
> > flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FileChannel.java, line 131
> > <https://reviews.apache.org/r/8899/diff/7/?file=270824#file270824line131>
> >
> >     As opposed to allowing this to be null and complicating the check below, how about passing in "" as a default and calling trim() on the returned value?

Will do.
> On March 21, 2013, 3:01 a.m., Brock Noland wrote:
> > flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Log.java, line 1044
> > <https://reviews.apache.org/r/8899/diff/7/?file=270827#file270827line1044>
> >
> >     I think "filesToDelete" would be better named "pendingDeletes" or something like that.

Will do.
> On March 21, 2013, 3:01 a.m., Brock Noland wrote:
> > flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Log.java, line 1068
> > <https://reviews.apache.org/r/8899/diff/7/?file=270827#file270827line1068>
> >
> >     Why not have a single code path? It will change the current behavior slightly but I don't see a huge issue with that.

Do you mean that we can hold on to the files till the next checkpoint even if dual checkpoints is disabled? That makes sense. I will do that in the next patch
- Hari
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/8899/#review18196
-----------------------------------------------------------
On March 14, 2013, 11:43 p.m., Hari Shreedharan wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/8899/
> -----------------------------------------------------------
>
> (Updated March 14, 2013, 11:43 p.m.)
>
>
> Review request for Flume.
>
>
> Description
> -------
>
> Added support for backup and retrieval of checkpoint.
>
>
> This addresses bug FLUME-1516.
>     https://issues.apache.org/jira/browse/FLUME-1516
+
Hari Shreedharan 2013-03-25, 21:03
+
Brock Noland 2013-03-25, 21:51
+
Hari Shreedharan 2013-03-25, 22:08
+
Brock Noland 2013-03-25, 22:12
+
Hari Shreedharan 2013-03-26, 22:52
+
Brock Noland 2013-04-04, 18:20
+
Hari Shreedharan 2013-04-04, 21:34
+
Hari Shreedharan 2013-04-04, 21:32
+
Hari Shreedharan 2013-04-05, 00:16
+
Brock Noland 2013-04-05, 14:35
+
Hari Shreedharan 2013-04-05, 18:01
+
Brock Noland 2013-04-05, 18:26