Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Flume >> mail # dev >> Review Request: FLUME-1586. File Channel should support verifying integrity of individual events.


Copy link to this message
-
Re: Review Request: FLUME-1586. File Channel should support verifying integrity of individual events.

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/10944/#review20961
-----------------------------------------------------------

flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/LogFile.java
<https://reviews.apache.org/r/10944/#comment43311>

    This byte represents an event which was detected as corrupt, and then replaced by the tool. Basically the idea is that if this is seen the reads should simply ignore this event (log and ignore).

flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/LogFile.java
<https://reviews.apache.org/r/10944/#comment43312>

    Will drop the "this." in the next patch.

flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/LogFile.java
<https://reviews.apache.org/r/10944/#comment43313>

    Will fix

flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/LogFile.java
<https://reviews.apache.org/r/10944/#comment43316>

    Will fix.

flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Serialization.java
<https://reviews.apache.org/r/10944/#comment43317>

    Intellij craziness. Will fix.

flume-tools/src/main/java/org/apache/flume/tools/FileChannelIntegrityTool.java
<https://reviews.apache.org/r/10944/#comment43318>

    There should be none. WIll remove the if.
- Hari Shreedharan
On May 23, 2013, 8:54 p.m., Hari Shreedharan wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/10944/
> -----------------------------------------------------------
>
> (Updated May 23, 2013, 8:54 p.m.)
>
>
> Review request for Flume.
>
>
> Description
> -------
>
> Patch to add a checksum to events and replace them with a noop event using a tool, if corrupt.
>
>
> This addresses bug FLUME-1586.
>     https://issues.apache.org/jira/browse/FLUME-1586
>
>
> Diffs
> -----
>
>   flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/CorruptEventException.java PRE-CREATION
>   flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FileChannel.java cc0d38a
>   flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Log.java 1918baa
>   flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/LogFile.java d3db896
>   flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/LogFileV3.java d9a2a9b
>   flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/NoopRecordException.java PRE-CREATION
>   flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Put.java 4235a79
>   flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/ReplayHandler.java fc47b23
>   flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Serialization.java d6897e1
>   flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/TransactionEventRecord.java 073042f
>   flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/proto/ProtosFactory.java 4860ac2
>   flume-ng-channels/flume-file-channel/src/main/proto/filechannel.proto 1e668d2
>   flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFileChannel.java 0f7d14d
>   flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestLog.java 54978f8
>   flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestLogFile.java bef22ef
>   flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestTransactionEventRecordV3.java f403422
>   flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestUtils.java 563dbcc
>   flume-ng-dist/pom.xml 1a44626