Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Flume >> mail # dev >> Review Request: FLUME-1586. File Channel should support verifying integrity of individual events.


Copy link to this message
-
Review Request: FLUME-1586. File Channel should support verifying integrity of individual events.

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/10944/
-----------------------------------------------------------

Review request for Flume.
Description
-------

Patch to add a checksum to events and replace them with a noop event using a tool, if corrupt.
This addresses bug FLUME-1586.
    https://issues.apache.org/jira/browse/FLUME-1586
Diffs
-----

  flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/CorruptEventException.java PRE-CREATION
  flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FileChannel.java a7aa70c
  flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Log.java e61437d
  flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/LogFile.java d3db896
  flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/LogFileV3.java d9a2a9b
  flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/NoopEventRecordException.java PRE-CREATION
  flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Put.java 4235a79
  flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/ReplayHandler.java fc47b23
  flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Serialization.java d6897e1
  flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/TransactionEventRecord.java 073042f
  flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/proto/ProtosFactory.java 4860ac2
  flume-ng-channels/flume-file-channel/src/main/proto/filechannel.proto 1e668d2
  flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFileChannel.java 0f7d14d
  flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestLog.java 54978f8
  flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestLogFile.java bef22ef
  flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestTransactionEventRecordV3.java f403422
  flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestUtils.java 563dbcc
  flume-tools/pom.xml PRE-CREATION
  flume-tools/src/main/java/org/apache/flume/tools/FileChannelIntegrityTool.java PRE-CREATION
  flume-tools/src/main/java/org/apache/flume/tools/FlumeTool.java PRE-CREATION
  flume-tools/src/main/java/org/apache/flume/tools/FlumeToolType.java PRE-CREATION
  flume-tools/src/main/java/org/apache/flume/tools/FlumeToolsMain.java PRE-CREATION
  pom.xml 686ecb6

Diff: https://reviews.apache.org/r/10944/diff/
Testing
-------

Added unit tests when corrupt and noop events are encountered. I will add tests for the tool as well soon. I have not yet tested the tool completely. This patch aims at gathering feedback on the approach.
Thanks,

Hari Shreedharan