Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Flume >> mail # dev >> Review Request: FLUME-1586. File Channel should support verifying integrity of individual events.


+
Hari Shreedharan 2013-05-06, 07:49
+
Hari Shreedharan 2013-05-06, 20:12
+
Hari Shreedharan 2013-05-06, 21:28
+
Hari Shreedharan 2013-05-23, 01:52
+
Hari Shreedharan 2013-05-23, 19:59
+
Hari Shreedharan 2013-05-23, 20:54
+
Brock Noland 2013-05-23, 22:01
Copy link to this message
-
Re: Review Request: FLUME-1586. File Channel should support verifying integrity of individual events.


> On May 23, 2013, 10:01 p.m., Brock Noland wrote:
> > flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/LogFile.java, line 60
> > <https://reviews.apache.org/r/10944/diff/6/?file=295919#file295919line60>
> >
> >     Curious as to what byte this ends up representing?

Oh, I guess I did not read this correctly earlier. The value of this byte is 0.
- Hari
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/10944/#review20956
-----------------------------------------------------------
On May 23, 2013, 11:14 p.m., Hari Shreedharan wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/10944/
> -----------------------------------------------------------
>
> (Updated May 23, 2013, 11:14 p.m.)
>
>
> Review request for Flume.
>
>
> Description
> -------
>
> Patch to add a checksum to events and replace them with a noop event using a tool, if corrupt.
>
>
> This addresses bug FLUME-1586.
>     https://issues.apache.org/jira/browse/FLUME-1586
>
>
> Diffs
> -----
>
>   bin/flume-ng 22b95b8
>   flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/CorruptEventException.java PRE-CREATION
>   flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FileChannel.java cc0d38a
>   flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Log.java 1918baa
>   flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/LogFile.java d3db896
>   flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/LogFileV3.java d9a2a9b
>   flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/NoopRecordException.java PRE-CREATION
>   flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Put.java 4235a79
>   flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/ReplayHandler.java fc47b23
>   flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Serialization.java d6897e1
>   flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/TransactionEventRecord.java 073042f
>   flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/proto/ProtosFactory.java 4860ac2
>   flume-ng-channels/flume-file-channel/src/main/proto/filechannel.proto 1e668d2
>   flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFileChannel.java 0f7d14d
>   flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestLog.java 54978f8
>   flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestLogFile.java bef22ef
>   flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestTransactionEventRecordV3.java f403422
>   flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestUtils.java 563dbcc
>   flume-ng-dist/pom.xml 1a44626
>   flume-ng-dist/src/main/assembly/bin.xml b6a8c8a
>   flume-ng-dist/src/main/assembly/src.xml 7fafab8
>   flume-tools/pom.xml PRE-CREATION
>   flume-tools/src/main/java/org/apache/flume/tools/FileChannelIntegrityTool.java PRE-CREATION
>   flume-tools/src/main/java/org/apache/flume/tools/FlumeTool.java PRE-CREATION
>   flume-tools/src/main/java/org/apache/flume/tools/FlumeToolType.java PRE-CREATION
>   flume-tools/src/main/java/org/apache/flume/tools/FlumeToolsMain.java PRE-CREATION
>   flume-tools/src/test/java/org/apache/flume/tools/TestFileChannelIntegrityTool.java PRE-CREATION
>   flume-tools/src/test/java/org/apache/flume/tools/TestFlumeToolsMain.java PRE-CREATION
>   pom.xml a6992f6
>
> Diff: https://reviews.apache.org/r/10944/diff/
>
>
> Testing
> -------
>
> Added unit tests when corrupt and noop events are encountered. I will add tests for the tool as well soon. I have not yet tested the tool completely. This patch aims at gathering feedback on the approach.
+
Hari Shreedharan 2013-05-23, 22:48
+
Hari Shreedharan 2013-05-23, 23:14
+
Hari Shreedharan 2013-05-24, 00:00
+
Hari Shreedharan 2013-05-24, 04:59
+
Brock Noland 2013-06-12, 05:18
+
Hari Shreedharan 2013-06-12, 18:51