Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Flume, mail # dev - Review Request: FLUME-1632: Persist progress on each file in file spooling client/source


+
Mike Percy 2012-12-14, 09:56
+
Brock Noland 2012-12-14, 15:22
+
Mike Percy 2012-12-16, 13:31
+
Mike Percy 2012-12-16, 13:31
+
Mike Percy 2012-12-16, 13:33
+
Mike Percy 2012-12-17, 08:58
+
Brock Noland 2012-12-18, 17:44
+
Mike Percy 2012-12-18, 21:25
+
Brock Noland 2012-12-18, 21:29
+
Mike Percy 2012-12-18, 21:50
+
Mike Percy 2012-12-18, 23:22
+
Brock Noland 2012-12-18, 23:42
+
Brock Noland 2012-12-19, 00:00
+
Brock Noland 2012-12-19, 01:54
+
Mike Percy 2012-12-19, 12:41
Copy link to this message
-
Re: Review Request: FLUME-1632: Persist progress on each file in file spooling client/source
Mike Percy 2012-12-19, 12:38

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/8596/
-----------------------------------------------------------

(Updated Dec. 19, 2012, 12:37 p.m.)
Review request for Flume.
Changes
-------

Fixed several concerns raised in the review
Description
-------

Defines EventDeserializer interface and uses it from the spooling source. Progress is persisted as bytes are read from the underlying file.
This addresses bug FLUME-1632.
    https://issues.apache.org/jira/browse/FLUME-1632
Diffs (updated)
-----

  flume-ng-core/pom.xml 0224519
  flume-ng-core/src/main/avro/TransferStateFileMeta.avsc PRE-CREATION
  flume-ng-core/src/main/java/org/apache/flume/client/avro/AvroCLIClient.java 37e9ffa
  flume-ng-core/src/main/java/org/apache/flume/client/avro/BufferedLineReader.java 718e1b2
  flume-ng-core/src/main/java/org/apache/flume/client/avro/EventReader.java PRE-CREATION
  flume-ng-core/src/main/java/org/apache/flume/client/avro/LineReader.java 904f22c
  flume-ng-core/src/main/java/org/apache/flume/client/avro/ReliableEventReader.java PRE-CREATION
  flume-ng-core/src/main/java/org/apache/flume/client/avro/ReliableSpoolingFileEventReader.java PRE-CREATION
  flume-ng-core/src/main/java/org/apache/flume/client/avro/SimpleTextLineEventReader.java PRE-CREATION
  flume-ng-core/src/main/java/org/apache/flume/client/avro/SpoolingFileLineReader.java 8362299
  flume-ng-core/src/main/java/org/apache/flume/serialization/DurablePositionTracker.java PRE-CREATION
  flume-ng-core/src/main/java/org/apache/flume/serialization/EventDeserializer.java PRE-CREATION
  flume-ng-core/src/main/java/org/apache/flume/serialization/EventDeserializerFactory.java PRE-CREATION
  flume-ng-core/src/main/java/org/apache/flume/serialization/EventDeserializerType.java PRE-CREATION
  flume-ng-core/src/main/java/org/apache/flume/serialization/EventSerDe.java PRE-CREATION
  flume-ng-core/src/main/java/org/apache/flume/serialization/EventSerializer.java a418935
  flume-ng-core/src/main/java/org/apache/flume/serialization/EventSerializerFactory.java 75853a9
  flume-ng-core/src/main/java/org/apache/flume/serialization/EventSerializerType.java afe8ed8
  flume-ng-core/src/main/java/org/apache/flume/serialization/LineDeserializer.java PRE-CREATION
  flume-ng-core/src/main/java/org/apache/flume/serialization/PositionTracker.java PRE-CREATION
  flume-ng-core/src/main/java/org/apache/flume/serialization/Resettable.java PRE-CREATION
  flume-ng-core/src/main/java/org/apache/flume/serialization/ResettableFileInputStream.java PRE-CREATION
  flume-ng-core/src/main/java/org/apache/flume/serialization/ResettableInputStream.java PRE-CREATION
  flume-ng-core/src/main/java/org/apache/flume/source/SpoolDirectorySource.java 61824d8
  flume-ng-core/src/main/java/org/apache/flume/source/SpoolDirectorySourceConfigurationConstants.java 806a661
  flume-ng-core/src/main/java/org/apache/flume/tools/PlatformDetect.java PRE-CREATION
  flume-ng-core/src/test/java/org/apache/flume/client/avro/TestBufferedLineReader.java 169abe5
  flume-ng-core/src/test/java/org/apache/flume/client/avro/TestSpoolingFileLineReader.java 740bc98
  flume-ng-core/src/test/java/org/apache/flume/serialization/ResettableTestStringInputStream.java PRE-CREATION
  flume-ng-core/src/test/java/org/apache/flume/serialization/TestDurablePositionTracker.java PRE-CREATION
  flume-ng-core/src/test/java/org/apache/flume/serialization/TestLineDeserializer.java PRE-CREATION
  flume-ng-core/src/test/java/org/apache/flume/serialization/TestReliableSpoolingFileEventReader.java PRE-CREATION
  flume-ng-core/src/test/java/org/apache/flume/serialization/TestResettableFileInputStream.java PRE-CREATION
  flume-ng-core/src/test/java/org/apache/flume/source/TestSpoolDirectorySource.java 6e87b21
  flume-ng-core/src/test/resources/TestResettableFileInputStream_1.avro PRE-CREATION
  flume-ng-core/src/test/resources/TestResettableFileInputStream_1.truncated.avro PRE-CREATION
  pom.xml b934c1d

Diff: https://reviews.apache.org/r/8596/diff/
Testing

Unit tests pass.
Thanks,

Mike Percy
+
Brock Noland 2012-12-19, 15:35
+
Brock Noland 2012-12-19, 17:40
+
Mike Percy 2012-12-20, 03:10
+
Brock Noland 2012-12-19, 18:12