Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Flume, mail # dev - Review Request: FLUME-1632: Persist progress on each file in file spooling client/source


Copy link to this message
-
Review Request: FLUME-1632: Persist progress on each file in file spooling client/source
Mike Percy 2012-12-14, 09:56

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/8596/
-----------------------------------------------------------

Review request for Flume.
Description
-------

Defines EventDeserializer interface and uses it from the spooling source. Progress is persisted as bytes are read from the underlying file.
This addresses bug FLUME-1632.
    https://issues.apache.org/jira/browse/FLUME-1632
Diffs
-----

  flume-ng-clients/flume-ng-forwarder/pom.xml PRE-CREATION
  flume-ng-clients/flume-ng-forwarder/src/main/java/org/apache/flume/clients/forwarder/FileClient.java PRE-CREATION
  flume-ng-clients/flume-ng-forwarder/src/main/java/org/apache/flume/clients/forwarder/FlumeTool.java PRE-CREATION
  flume-ng-clients/flume-ng-forwarder/src/main/java/org/apache/flume/clients/forwarder/ToolRegistry.java PRE-CREATION
  flume-ng-clients/flume-ng-forwarder/src/main/java/org/apache/flume/clients/forwarder/ToolRunner.java PRE-CREATION
  flume-ng-clients/flume-ng-forwarder/src/test/java/org/apache/flume/clients/forwarder/Blah2.java PRE-CREATION
  flume-ng-clients/flume-ng-forwarder/src/test/resources/flume-log4jtest.properties PRE-CREATION
  flume-ng-clients/pom.xml 7ea2d1d
  flume-ng-core/pom.xml 2554a0e
  flume-ng-core/src/main/avro/TransferStateFileMeta.avsc PRE-CREATION
  flume-ng-core/src/main/java/org/apache/flume/client/avro/AvroCLIClient.java 37e9ffa
  flume-ng-core/src/main/java/org/apache/flume/client/avro/SpoolingFileEventParser.java PRE-CREATION
  flume-ng-core/src/main/java/org/apache/flume/client/avro/SpoolingFileLineReader.java 8362299
  flume-ng-core/src/main/java/org/apache/flume/serialization/EventDeserializer.java PRE-CREATION
  flume-ng-core/src/main/java/org/apache/flume/serialization/EventDeserializerFactory.java PRE-CREATION
  flume-ng-core/src/main/java/org/apache/flume/serialization/EventDeserializerType.java PRE-CREATION
  flume-ng-core/src/main/java/org/apache/flume/serialization/EventSerDe.java PRE-CREATION
  flume-ng-core/src/main/java/org/apache/flume/serialization/EventSerializer.java a418935
  flume-ng-core/src/main/java/org/apache/flume/serialization/LineDeserializer.java PRE-CREATION
  flume-ng-core/src/main/java/org/apache/flume/serialization/PositionTracker.java PRE-CREATION
  flume-ng-core/src/main/java/org/apache/flume/serialization/PositionTrackerFile.java PRE-CREATION
  flume-ng-core/src/main/java/org/apache/flume/serialization/Resettable.java PRE-CREATION
  flume-ng-core/src/main/java/org/apache/flume/serialization/ResettableFileInputStream.java PRE-CREATION
  flume-ng-core/src/main/java/org/apache/flume/serialization/ResettableInputStream.java PRE-CREATION
  flume-ng-core/src/main/java/org/apache/flume/serialization/Seekable.java PRE-CREATION
  flume-ng-core/src/main/java/org/apache/flume/source/SpoolDirectorySource.java 61824d8
  flume-ng-core/src/main/java/org/apache/flume/source/SpoolDirectorySourceConfigurationConstants.java 806a661
  flume-ng-core/src/test/java/org/apache/flume/client/avro/TestSpoolingFileLineReader.java 740bc98
  flume-ng-core/src/test/java/org/apache/flume/serialization/TestResettableFileInputStream.java PRE-CREATION
  flume-ng-core/src/test/java/org/apache/flume/source/TestSpoolDirectorySource.java 6e87b21
  flume-ng-core/src/test/resources/TestResettableFileInputStream_1.avro PRE-CREATION
  flume-ng-core/src/test/resources/TestResettableFileInputStream_1.truncated.avro PRE-CREATION
  pom.xml 53ac96b

Diff: https://reviews.apache.org/r/8596/diff/
Testing
-------

Early patch. Not all unit tests are passing yet. I am posting this for early feedback.
Thanks,

Mike Percy

+
Brock Noland 2012-12-14, 15:22
+
Mike Percy 2012-12-16, 13:31
+
Mike Percy 2012-12-16, 13:31
+
Mike Percy 2012-12-16, 13:33
+
Mike Percy 2012-12-17, 08:58
+
Brock Noland 2012-12-18, 17:44
+
Mike Percy 2012-12-18, 21:25
+
Brock Noland 2012-12-18, 21:29
+
Mike Percy 2012-12-18, 21:50
+
Mike Percy 2012-12-18, 23:22
+
Brock Noland 2012-12-18, 23:42
+
Brock Noland 2012-12-19, 00:00
+
Brock Noland 2012-12-19, 01:54
+
Mike Percy 2012-12-19, 12:41
+
Mike Percy 2012-12-19, 12:38
+
Brock Noland 2012-12-19, 15:35
+
Brock Noland 2012-12-19, 17:40
+
Mike Percy 2012-12-20, 03:10
+
Brock Noland 2012-12-19, 18:12