Flume, mail # dev - Review Request: FLUME-1763: FileChannel Checkpoints should not be done without free space

Review Request: FLUME-1763: FileChannel Checkpoints should not be done without free space

Brock Noland 2012-12-06, 21:52

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/8379/
-----------------------------------------------------------

Review request for Flume.
Description
-------

Adds a parameter which can be configured by defaults to 500MB. We will refuse to checkpoint or perform log writes in this case.
This addresses bug FLUME-1763.
   https://issues.apache.org/jira/browse/FLUME-1763
Diffs
-----

  flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FileChannel.java 64a1350
  flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FileChannelConfiguration.java 92cad77
  flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Log.java 6d1cf51
  flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestLog.java a165d6a
  flume-ng-doc/sphinx/FlumeUserGuide.rst f8528bb

Diff: https://reviews.apache.org/r/8379/diff/
Testing
-------

Two unit tests added.
Thanks,

Brock Noland

Re: Review Request: FLUME-1763: FileChannel Checkpoints should not be done without free space

Brock Noland 2012-12-06, 21:54

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/8379/
-----------------------------------------------------------

(Updated Dec. 6, 2012, 9:54 p.m.)
Review request for Flume.
Changes
-------

Removed whitespace
Description
-------

Adds a parameter which can be configured by defaults to 500MB. We will refuse to checkpoint or perform log writes in this case.
This addresses bug FLUME-1763.
   https://issues.apache.org/jira/browse/FLUME-1763
Diffs (updated)
-----

  flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FileChannel.java 64a1350
  flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FileChannelConfiguration.java 92cad77
  flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Log.java 6d1cf51
  flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestLog.java a165d6a
  flume-ng-doc/sphinx/FlumeUserGuide.rst f8528bb

Diff: https://reviews.apache.org/r/8379/diff/
Testing
-------

Two unit tests added.
Thanks,

Brock Noland

Re: Review Request: FLUME-1763: FileChannel Checkpoints should not be done without free space

Brock Noland 2012-12-06, 22:02

Re: Review Request: FLUME-1763: FileChannel Checkpoints should not be done without free space

Brock Noland 2012-12-06, 22:03

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/8379/
-----------------------------------------------------------

(Updated Dec. 6, 2012, 10:03 p.m.)
Review request for Flume.
Changes
-------

Fixed self review comment.
Description
-------

Adds a parameter which can be configured by defaults to 500MB. We will refuse to checkpoint or perform log writes in this case.
This addresses bug FLUME-1763.
   https://issues.apache.org/jira/browse/FLUME-1763
Diffs (updated)
-----

  flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FileChannel.java 64a1350
  flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FileChannelConfiguration.java 92cad77
  flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Log.java 6d1cf51
  flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestLog.java a165d6a
  flume-ng-doc/sphinx/FlumeUserGuide.rst f8528bb

Diff: https://reviews.apache.org/r/8379/diff/
Testing
-------

Two unit tests added.
Thanks,

Brock Noland

Re: Review Request: FLUME-1763: FileChannel Checkpoints should not be done without free space

Brock Noland 2012-12-10, 23:28

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/8379/
-----------------------------------------------------------

(Updated Dec. 10, 2012, 11:28 p.m.)
Review request for Flume.
Changes
-------

Fixed bug Hari found.
Description
-------

Adds a parameter which can be configured by defaults to 500MB. We will refuse to checkpoint or perform log writes in this case.
This addresses bug FLUME-1763.
   https://issues.apache.org/jira/browse/FLUME-1763
Diffs (updated)
-----

  flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FileChannel.java 64a1350
  flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FileChannelConfiguration.java 92cad77
  flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Log.java 6d1cf51
  flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestLog.java a165d6a
  flume-ng-doc/sphinx/FlumeUserGuide.rst 21ca5cc

Diff: https://reviews.apache.org/r/8379/diff/
Testing
-------

Two unit tests added.
Thanks,

Brock Noland

Re: Review Request: FLUME-1763: FileChannel Checkpoints should not be done without free space

Brock Noland 2012-12-11, 00:45

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/8379/
-----------------------------------------------------------

(Updated Dec. 11, 2012, 12:45 a.m.)
Review request for Flume.
Changes
-------

Removed extra Math.max in log class.
Description
-------

Adds a parameter which can be configured by defaults to 500MB. We will refuse to checkpoint or perform log writes in this case.
This addresses bug FLUME-1763.
   https://issues.apache.org/jira/browse/FLUME-1763
Diffs (updated)
-----

  flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FileChannel.java 64a1350
  flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FileChannelConfiguration.java 92cad77
  flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Log.java 6d1cf51
  flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestLog.java a165d6a
  flume-ng-doc/sphinx/FlumeUserGuide.rst 21ca5cc

Diff: https://reviews.apache.org/r/8379/diff/
Testing
-------

Two unit tests added.
Thanks,

Brock Noland

Re: Review Request: FLUME-1763: FileChannel Checkpoints should not be done without free space

Hari Shreedharan 2012-12-11, 01:04

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/8379/#review14280
-----------------------------------------------------------

Ship it!
Ship It!

- Hari Shreedharan
On Dec. 11, 2012, 12:45 a.m., Brock Noland wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/8379/
> -----------------------------------------------------------
>
> (Updated Dec. 11, 2012, 12:45 a.m.)
>
>
> Review request for Flume.
>
>
> Description
> -------
>
> Adds a parameter which can be configured by defaults to 500MB. We will refuse to checkpoint or perform log writes in this case.
>
>
> This addresses bug FLUME-1763.
> https://issues.apache.org/jira/browse/FLUME-1763
>
>
> Diffs
> -----
>
> flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FileChannel.java 64a1350
> flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FileChannelConfiguration.java 92cad77
> flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Log.java 6d1cf51
> flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestLog.java a165d6a
> flume-ng-doc/sphinx/FlumeUserGuide.rst 21ca5cc
>
> Diff: https://reviews.apache.org/r/8379/diff/
>
>
> Testing
> -------
>
> Two unit tests added.
>
>
> Thanks,
>
> Brock Noland
>
>

Re: Review Request: FLUME-1763: FileChannel Checkpoints should not be done without free space

Hari Shreedharan 2012-12-10, 23:36

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/8379/#review14272
-----------------------------------------------------------

Ship it!
Looks good. I added one really minor comment below. Feel free to ignore. When you are ready, please attach patch to jira.
flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Log.java
<https://reviews.apache.org/r/8379/#comment30408>

Isn't it always guaranteed that minimumRequiredSpace is >= FileChannelConfiguration.FLOOR_MINIMUM_REQUIRED_SPACE since the same comparison was made earlier in FileChannel.java?
- Hari Shreedharan
On Dec. 10, 2012, 11:28 p.m., Brock Noland wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/8379/
> -----------------------------------------------------------
>
> (Updated Dec. 10, 2012, 11:28 p.m.)
>
>
> Review request for Flume.
>
>
> Description
> -------
>
> Adds a parameter which can be configured by defaults to 500MB. We will refuse to checkpoint or perform log writes in this case.
>
>
> This addresses bug FLUME-1763.
> https://issues.apache.org/jira/browse/FLUME-1763
>
>
> Diffs
> -----
>
> flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FileChannel.java 64a1350
> flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FileChannelConfiguration.java 92cad77
> flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Log.java 6d1cf51
> flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestLog.java a165d6a
> flume-ng-doc/sphinx/FlumeUserGuide.rst 21ca5cc
>
> Diff: https://reviews.apache.org/r/8379/diff/
>
>
> Testing
> -------
>
> Two unit tests added.
>
>
> Thanks,
>
> Brock Noland
>
>

Re: Review Request: FLUME-1763: FileChannel Checkpoints should not be done without free space

Hari Shreedharan 2012-12-10, 21:54

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/8379/#review14267
-----------------------------------------------------------
Generally looks good, but I am confused about the configuration code.
flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FileChannel.java
<https://reviews.apache.org/r/8379/#comment30405>

Why do this when you are taking the max of this value and FileChannelConfiguration.FLOOR_MINIMUM_REQUIRED_SPACE later? Did you want to do a Math.max here?

flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Log.java
<https://reviews.apache.org/r/8379/#comment30404>

Looks like you are checking for min of the configured value and FileChannelConfiguration.FLOOR_MINIMUM_REQUIRED_SPACE in FileChannel.java and you are checking the max of that value and FileChannelConfiguration.FLOOR_MINIMUM_REQUIRED_SPACE. Doing both these together means that minimumRequiredSpace is always equal to FileChannelConfiguration.FLOOR_MINIMUM_REQUIRED_SPACE. Am I missing something here?
- Hari Shreedharan
On Dec. 6, 2012, 10:03 p.m., Brock Noland wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/8379/
> -----------------------------------------------------------
>
> (Updated Dec. 6, 2012, 10:03 p.m.)
>
>
> Review request for Flume.
>
>
> Description
> -------
>
> Adds a parameter which can be configured by defaults to 500MB. We will refuse to checkpoint or perform log writes in this case.
>
>
> This addresses bug FLUME-1763.
> https://issues.apache.org/jira/browse/FLUME-1763
>
>
> Diffs
> -----
>
> flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FileChannel.java 64a1350
> flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FileChannelConfiguration.java 92cad77
> flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Log.java 6d1cf51
> flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestLog.java a165d6a
> flume-ng-doc/sphinx/FlumeUserGuide.rst f8528bb
>
> Diff: https://reviews.apache.org/r/8379/diff/
>
>
> Testing
> -------
>
> Two unit tests added.
>
>
> Thanks,
>
> Brock Noland
>
>

Re: Review Request: FLUME-1763: FileChannel Checkpoints should not be done without free space

Brock Noland 2012-12-10, 23:29

> On Dec. 10, 2012, 9:54 p.m., Hari Shreedharan wrote:
> > flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Log.java, lines 236-237
> > <https://reviews.apache.org/r/8379/diff/3/?file=235050#file235050line236>
> >
> > Looks like you are checking for min of the configured value and FileChannelConfiguration.FLOOR_MINIMUM_REQUIRED_SPACE in FileChannel.java and you are checking the max of that value and FileChannelConfiguration.FLOOR_MINIMUM_REQUIRED_SPACE. Doing both these together means that minimumRequiredSpace is always equal to FileChannelConfiguration.FLOOR_MINIMUM_REQUIRED_SPACE. Am I missing something here?

Good catch! We should have been taking the max in FileChannel.
- Brock
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/8379/#review14267
-----------------------------------------------------------
On Dec. 10, 2012, 11:28 p.m., Brock Noland wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/8379/
> -----------------------------------------------------------
>
> (Updated Dec. 10, 2012, 11:28 p.m.)
>
>
> Review request for Flume.
>
>
> Description
> -------
>
> Adds a parameter which can be configured by defaults to 500MB. We will refuse to checkpoint or perform log writes in this case.
>
>
> This addresses bug FLUME-1763.
> https://issues.apache.org/jira/browse/FLUME-1763
>
>
> Diffs
> -----
>
> flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FileChannel.java 64a1350
> flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FileChannelConfiguration.java 92cad77
> flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Log.java 6d1cf51
> flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestLog.java a165d6a
> flume-ng-doc/sphinx/FlumeUserGuide.rst 21ca5cc
>
> Diff: https://reviews.apache.org/r/8379/diff/
>
>
> Testing
> -------
>
> Two unit tests added.
>
>
> Thanks,
>
> Brock Noland
>
>