Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Flume, mail # dev - Review Request: FLUME-1586. File Channel should support verifying integrity of individual events.


+
Hari Shreedharan 2013-05-06, 07:49
+
Hari Shreedharan 2013-05-06, 20:12
+
Hari Shreedharan 2013-05-06, 21:28
Copy link to this message
-
Re: Review Request: FLUME-1586. File Channel should support verifying integrity of individual events.
Hari Shreedharan 2013-05-23, 01:52

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/10944/
-----------------------------------------------------------

(Updated May 23, 2013, 1:52 a.m.)
Review request for Flume.
Changes
-------

Add tests + refactoring.
Description
-------

Patch to add a checksum to events and replace them with a noop event using a tool, if corrupt.
This addresses bug FLUME-1586.
    https://issues.apache.org/jira/browse/FLUME-1586
Diffs (updated)
-----

  flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/CorruptEventException.java PRE-CREATION
  flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FileChannel.java cc0d38a
  flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FlumeEvent.java c447335
  flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FlumeEventPointer.java 5f06ab7
  flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Log.java 1918baa
  flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/LogFile.java d3db896
  flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/LogFileV3.java d9a2a9b
  flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/NoopRecordException.java PRE-CREATION
  flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Pair.java dfcdd73
  flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Put.java 4235a79
  flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/ReplayHandler.java fc47b23
  flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Serialization.java d6897e1
  flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/TransactionEventRecord.java 073042f
  flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/proto/ProtosFactory.java 4860ac2
  flume-ng-channels/flume-file-channel/src/main/proto/filechannel.proto 1e668d2
  flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFileChannel.java 0f7d14d
  flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestLog.java 54978f8
  flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestLogFile.java bef22ef
  flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestTransactionEventRecordV3.java f403422
  flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestUtils.java 563dbcc
  flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/AbstractHDFSWriter.java bc3b383
  flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSCompressedDataStream.java 2c2be6a
  flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSDataStream.java b8214be
  flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSSequenceFile.java 0383744
  flume-tools/pom.xml PRE-CREATION
  flume-tools/src/main/java/org/apache/flume/tools/FileChannelIntegrityTool.java PRE-CREATION
  flume-tools/src/main/java/org/apache/flume/tools/FlumeTool.java PRE-CREATION
  flume-tools/src/main/java/org/apache/flume/tools/FlumeToolType.java PRE-CREATION
  flume-tools/src/main/java/org/apache/flume/tools/FlumeToolsMain.java PRE-CREATION
  flume-tools/src/test/java/org/apache/flume/tools/TestFileChannelIntegrityTool.java PRE-CREATION
  flume-tools/src/test/java/org/apache/flume/tools/TestFlumeToolsMain.java PRE-CREATION
  pom.xml a6992f6

Diff: https://reviews.apache.org/r/10944/diff/
Testing
-------

Added unit tests when corrupt and noop events are encountered. I will add tests for the tool as well soon. I have not yet tested the tool completely. This patch aims at gathering feedback on the approach.
Thanks,

Hari Shreedharan
+
Hari Shreedharan 2013-05-23, 19:59
+
Hari Shreedharan 2013-05-23, 20:54
+
Brock Noland 2013-05-23, 22:01
+
Hari Shreedharan 2013-05-23, 23:52
+
Hari Shreedharan 2013-05-23, 22:48
+
Hari Shreedharan 2013-05-23, 23:14
+
Hari Shreedharan 2013-05-24, 00:00
+
Hari Shreedharan 2013-05-24, 04:59
+
Brock Noland 2013-06-12, 05:18
+
Hari Shreedharan 2013-06-12, 18:51