Flume, mail # dev - Re: Review Request: FLUME-1632: Persist progress on each file in file spooling client/source

Review Request: FLUME-1632: Persist progress on each file in file spooling client/source

Mike Percy 2012-12-14, 09:56

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/8596/
-----------------------------------------------------------

Review request for Flume.
Description
-------

Defines EventDeserializer interface and uses it from the spooling source. Progress is persisted as bytes are read from the underlying file.
This addresses bug FLUME-1632.
   https://issues.apache.org/jira/browse/FLUME-1632
Diffs
-----

  flume-ng-clients/flume-ng-forwarder/pom.xml PRE-CREATION
  flume-ng-clients/flume-ng-forwarder/src/main/java/org/apache/flume/clients/forwarder/FileClient.java PRE-CREATION
  flume-ng-clients/flume-ng-forwarder/src/main/java/org/apache/flume/clients/forwarder/FlumeTool.java PRE-CREATION
  flume-ng-clients/flume-ng-forwarder/src/main/java/org/apache/flume/clients/forwarder/ToolRegistry.java PRE-CREATION
  flume-ng-clients/flume-ng-forwarder/src/main/java/org/apache/flume/clients/forwarder/ToolRunner.java PRE-CREATION
  flume-ng-clients/flume-ng-forwarder/src/test/java/org/apache/flume/clients/forwarder/Blah2.java PRE-CREATION
  flume-ng-clients/flume-ng-forwarder/src/test/resources/flume-log4jtest.properties PRE-CREATION
  flume-ng-clients/pom.xml 7ea2d1d
  flume-ng-core/pom.xml 2554a0e
  flume-ng-core/src/main/avro/TransferStateFileMeta.avsc PRE-CREATION
  flume-ng-core/src/main/java/org/apache/flume/client/avro/AvroCLIClient.java 37e9ffa
  flume-ng-core/src/main/java/org/apache/flume/client/avro/SpoolingFileEventParser.java PRE-CREATION
  flume-ng-core/src/main/java/org/apache/flume/client/avro/SpoolingFileLineReader.java 8362299
  flume-ng-core/src/main/java/org/apache/flume/serialization/EventDeserializer.java PRE-CREATION
  flume-ng-core/src/main/java/org/apache/flume/serialization/EventDeserializerFactory.java PRE-CREATION
  flume-ng-core/src/main/java/org/apache/flume/serialization/EventDeserializerType.java PRE-CREATION
  flume-ng-core/src/main/java/org/apache/flume/serialization/EventSerDe.java PRE-CREATION
  flume-ng-core/src/main/java/org/apache/flume/serialization/EventSerializer.java a418935
  flume-ng-core/src/main/java/org/apache/flume/serialization/LineDeserializer.java PRE-CREATION
  flume-ng-core/src/main/java/org/apache/flume/serialization/PositionTracker.java PRE-CREATION
  flume-ng-core/src/main/java/org/apache/flume/serialization/PositionTrackerFile.java PRE-CREATION
  flume-ng-core/src/main/java/org/apache/flume/serialization/Resettable.java PRE-CREATION
  flume-ng-core/src/main/java/org/apache/flume/serialization/ResettableFileInputStream.java PRE-CREATION
  flume-ng-core/src/main/java/org/apache/flume/serialization/ResettableInputStream.java PRE-CREATION
  flume-ng-core/src/main/java/org/apache/flume/serialization/Seekable.java PRE-CREATION
  flume-ng-core/src/main/java/org/apache/flume/source/SpoolDirectorySource.java 61824d8
  flume-ng-core/src/main/java/org/apache/flume/source/SpoolDirectorySourceConfigurationConstants.java 806a661
  flume-ng-core/src/test/java/org/apache/flume/client/avro/TestSpoolingFileLineReader.java 740bc98
  flume-ng-core/src/test/java/org/apache/flume/serialization/TestResettableFileInputStream.java PRE-CREATION
  flume-ng-core/src/test/java/org/apache/flume/source/TestSpoolDirectorySource.java 6e87b21
  flume-ng-core/src/test/resources/TestResettableFileInputStream_1.avro PRE-CREATION
  flume-ng-core/src/test/resources/TestResettableFileInputStream_1.truncated.avro PRE-CREATION
  pom.xml 53ac96b

Diff: https://reviews.apache.org/r/8596/diff/
Testing
-------

Early patch. Not all unit tests are passing yet. I am posting this for early feedback.
Thanks,

Mike Percy

Re: Review Request: FLUME-1632: Persist progress on each file in file spooling client/source

Brock Noland 2012-12-14, 15:22

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/8596/#review14504
-----------------------------------------------------------
Mike,

Nice work!!
A few comments below. My only concern about the implementation is how often fsync() is being is being called. I don't know how often that will be and if it's often enough I could see the source being unable to keep up with input data.
flume-ng-clients/flume-ng-forwarder/src/main/java/org/apache/flume/clients/forwarder/FlumeTool.java
<https://reviews.apache.org/r/8596/#comment30880>

   Should this tool stuff be in core? Seems very useful for more than the forwarder...

flume-ng-clients/flume-ng-forwarder/src/main/java/org/apache/flume/clients/forwarder/ToolRunner.java
<https://reviews.apache.org/r/8596/#comment30879>

   Returns null if the name is not found

flume-ng-clients/flume-ng-forwarder/src/test/java/org/apache/flume/clients/forwarder/Blah2.java
<https://reviews.apache.org/r/8596/#comment30881>

   Probably a mistake?

flume-ng-core/src/main/java/org/apache/flume/client/avro/SpoolingFileEventParser.java
<https://reviews.apache.org/r/8596/#comment30882>

   As opposed to the note, could you use the interface annotations?

flume-ng-core/src/main/java/org/apache/flume/client/avro/SpoolingFileEventParser.java
<https://reviews.apache.org/r/8596/#comment30883>

   If this delete fails we will poison there data. I think we show throw an exception if it fails.

flume-ng-core/src/main/java/org/apache/flume/serialization/EventDeserializerFactory.java
<https://reviews.apache.org/r/8596/#comment30878>

   Do you think we could give a better error message here? That the class is not of Builder type?

flume-ng-core/src/main/java/org/apache/flume/serialization/LineDeserializer.java
<https://reviews.apache.org/r/8596/#comment30887>

   nit: args are reverse order as the builder

flume-ng-core/src/main/java/org/apache/flume/serialization/LineDeserializer.java
<https://reviews.apache.org/r/8596/#comment30884>

   readLine can return null.

flume-ng-core/src/main/java/org/apache/flume/serialization/PositionTrackerFile.java
<https://reviews.apache.org/r/8596/#comment30888>

   How often do we feel storePosition is going to be called? The reason I am wondering is that sync is pretty expensive.

flume-ng-core/src/test/java/org/apache/flume/client/avro/TestSpoolingFileLineReader.java
<https://reviews.apache.org/r/8596/#comment30885>

   How come all these tests are now ignored? -- nevermind I see the comment about this being an early patch

flume-ng-core/src/test/java/org/apache/flume/serialization/TestResettableFileInputStream.java
<https://reviews.apache.org/r/8596/#comment30886>

   Should be true by default? (ie false only in the ide of the developer?
- Brock Noland
On Dec. 14, 2012, 9:56 a.m., Mike Percy wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/8596/
> -----------------------------------------------------------
>
> (Updated Dec. 14, 2012, 9:56 a.m.)
>
>
> Review request for Flume.
>
>
> Description
> -------
>
> Defines EventDeserializer interface and uses it from the spooling source. Progress is persisted as bytes are read from the underlying file.
>
>
> This addresses bug FLUME-1632.
> https://issues.apache.org/jira/browse/FLUME-1632
>
>
> Diffs
> -----
>
> flume-ng-clients/flume-ng-forwarder/pom.xml PRE-CREATION
> flume-ng-clients/flume-ng-forwarder/src/main/java/org/apache/flume/clients/forwarder/FileClient.java PRE-CREATION
> flume-ng-clients/flume-ng-forwarder/src/main/java/org/apache/flume/clients/forwarder/FlumeTool.java PRE-CREATION
> flume-ng-clients/flume-ng-forwarder/src/main/java/org/apache/flume/clients/forwarder/ToolRegistry.java PRE-CREATION
> flume-ng-clients/flume-ng-forwarder/src/main/java/org/apache/flume/clients/forwarder/ToolRunner.java PRE-CREATION

Re: Review Request: FLUME-1632: Persist progress on each file in file spooling client/source

Mike Percy 2012-12-16, 13:31

> On Dec. 14, 2012, 3:22 p.m., Brock Noland wrote:
> > flume-ng-clients/flume-ng-forwarder/src/main/java/org/apache/flume/clients/forwarder/FlumeTool.java, line 1
> > <https://reviews.apache.org/r/8596/diff/1/?file=238552#file238552line1>
> >
> > Should this tool stuff be in core? Seems very useful for more than the forwarder...

I'm considering splitting this out into a separate patch soon, actually. It's not really baked.
> On Dec. 14, 2012, 3:22 p.m., Brock Noland wrote:
> > flume-ng-clients/flume-ng-forwarder/src/test/java/org/apache/flume/clients/forwarder/Blah2.java, line 1
> > <https://reviews.apache.org/r/8596/diff/1/?file=238555#file238555line1>
> >
> > Probably a mistake?

Placeholder for the tests.
- Mike
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/8596/#review14504
-----------------------------------------------------------
On Dec. 14, 2012, 9:56 a.m., Mike Percy wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/8596/
> -----------------------------------------------------------
>
> (Updated Dec. 14, 2012, 9:56 a.m.)
>
>
> Review request for Flume.
>
>
> Description
> -------
>
> Defines EventDeserializer interface and uses it from the spooling source. Progress is persisted as bytes are read from the underlying file.
>
>
> This addresses bug FLUME-1632.
> https://issues.apache.org/jira/browse/FLUME-1632
>
>
> Diffs
> -----
>
> flume-ng-clients/flume-ng-forwarder/pom.xml PRE-CREATION
> flume-ng-clients/flume-ng-forwarder/src/main/java/org/apache/flume/clients/forwarder/FileClient.java PRE-CREATION
> flume-ng-clients/flume-ng-forwarder/src/main/java/org/apache/flume/clients/forwarder/FlumeTool.java PRE-CREATION
> flume-ng-clients/flume-ng-forwarder/src/main/java/org/apache/flume/clients/forwarder/ToolRegistry.java PRE-CREATION
> flume-ng-clients/flume-ng-forwarder/src/main/java/org/apache/flume/clients/forwarder/ToolRunner.java PRE-CREATION
> flume-ng-clients/flume-ng-forwarder/src/test/java/org/apache/flume/clients/forwarder/Blah2.java PRE-CREATION
> flume-ng-clients/flume-ng-forwarder/src/test/resources/flume-log4jtest.properties PRE-CREATION
> flume-ng-clients/pom.xml 7ea2d1d
> flume-ng-core/pom.xml 2554a0e
> flume-ng-core/src/main/avro/TransferStateFileMeta.avsc PRE-CREATION
> flume-ng-core/src/main/java/org/apache/flume/client/avro/AvroCLIClient.java 37e9ffa
> flume-ng-core/src/main/java/org/apache/flume/client/avro/SpoolingFileEventParser.java PRE-CREATION
> flume-ng-core/src/main/java/org/apache/flume/client/avro/SpoolingFileLineReader.java 8362299
> flume-ng-core/src/main/java/org/apache/flume/serialization/EventDeserializer.java PRE-CREATION
> flume-ng-core/src/main/java/org/apache/flume/serialization/EventDeserializerFactory.java PRE-CREATION
> flume-ng-core/src/main/java/org/apache/flume/serialization/EventDeserializerType.java PRE-CREATION
> flume-ng-core/src/main/java/org/apache/flume/serialization/EventSerDe.java PRE-CREATION
> flume-ng-core/src/main/java/org/apache/flume/serialization/EventSerializer.java a418935
> flume-ng-core/src/main/java/org/apache/flume/serialization/LineDeserializer.java PRE-CREATION
> flume-ng-core/src/main/java/org/apache/flume/serialization/PositionTracker.java PRE-CREATION
> flume-ng-core/src/main/java/org/apache/flume/serialization/PositionTrackerFile.java PRE-CREATION
> flume-ng-core/src/main/java/org/apache/flume/serialization/Resettable.java PRE-CREATION
> flume-ng-core/src/main/java/org/apache/flume/serialization/ResettableFileInputStream.java PRE-CREATION
> flume-ng-core/src/main/java/org/apache/flume/serialization/ResettableInputStream.java PRE-CREATION
> flume-ng-core/src/main/java/org/apache/flume/serialization/Seekable.java PRE-CREATION
> flume-ng-core/src/main/java/org/apache/flume/source/SpoolDirectorySource.java 61824d8

Re: Review Request: FLUME-1632: Persist progress on each file in file spooling client/source

Mike Percy 2012-12-16, 13:33

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/8596/
-----------------------------------------------------------

(Updated Dec. 16, 2012, 1:33 p.m.)
Review request for Flume.
Changes
-------

Tests are passing. The one major thing I still need to do is I need to port the avro-client CLI tool to the new interface. Right now the avro-client is crippled.
Description
-------

Defines EventDeserializer interface and uses it from the spooling source. Progress is persisted as bytes are read from the underlying file.
This addresses bug FLUME-1632.
   https://issues.apache.org/jira/browse/FLUME-1632
Diffs (updated)
-----

  flume-ng-clients/flume-ng-forwarder/pom.xml PRE-CREATION
  flume-ng-clients/flume-ng-forwarder/src/main/java/org/apache/flume/clients/forwarder/FileClient.java PRE-CREATION
  flume-ng-clients/flume-ng-forwarder/src/main/java/org/apache/flume/clients/forwarder/FlumeTool.java PRE-CREATION
  flume-ng-clients/flume-ng-forwarder/src/main/java/org/apache/flume/clients/forwarder/ToolRegistry.java PRE-CREATION
  flume-ng-clients/flume-ng-forwarder/src/main/java/org/apache/flume/clients/forwarder/ToolRunner.java PRE-CREATION
  flume-ng-clients/flume-ng-forwarder/src/test/java/org/apache/flume/clients/forwarder/Blah2.java PRE-CREATION
  flume-ng-clients/flume-ng-forwarder/src/test/resources/flume-log4jtest.properties PRE-CREATION
  flume-ng-clients/pom.xml 7ea2d1d
  flume-ng-core/pom.xml 0224519
  flume-ng-core/src/main/avro/TransferStateFileMeta.avsc PRE-CREATION
  flume-ng-core/src/main/java/org/apache/flume/client/avro/AvroCLIClient.java 37e9ffa
  flume-ng-core/src/main/java/org/apache/flume/client/avro/SpoolingFileEventParser.java PRE-CREATION
  flume-ng-core/src/main/java/org/apache/flume/client/avro/SpoolingFileLineReader.java 8362299
  flume-ng-core/src/main/java/org/apache/flume/serialization/DurablePositionTracker.java PRE-CREATION
  flume-ng-core/src/main/java/org/apache/flume/serialization/EventDeserializer.java PRE-CREATION
  flume-ng-core/src/main/java/org/apache/flume/serialization/EventDeserializerFactory.java PRE-CREATION
  flume-ng-core/src/main/java/org/apache/flume/serialization/EventDeserializerType.java PRE-CREATION
  flume-ng-core/src/main/java/org/apache/flume/serialization/EventSerDe.java PRE-CREATION
  flume-ng-core/src/main/java/org/apache/flume/serialization/EventSerializer.java a418935
  flume-ng-core/src/main/java/org/apache/flume/serialization/EventSerializerFactory.java 75853a9
  flume-ng-core/src/main/java/org/apache/flume/serialization/LineDeserializer.java PRE-CREATION
  flume-ng-core/src/main/java/org/apache/flume/serialization/PositionTracker.java PRE-CREATION
  flume-ng-core/src/main/java/org/apache/flume/serialization/Resettable.java PRE-CREATION
  flume-ng-core/src/main/java/org/apache/flume/serialization/ResettableFileInputStream.java PRE-CREATION
  flume-ng-core/src/main/java/org/apache/flume/serialization/ResettableInputStream.java PRE-CREATION
  flume-ng-core/src/main/java/org/apache/flume/serialization/Seekable.java PRE-CREATION
  flume-ng-core/src/main/java/org/apache/flume/serialization/TransientPositionTracker.java PRE-CREATION
  flume-ng-core/src/main/java/org/apache/flume/source/SpoolDirectorySource.java 61824d8
  flume-ng-core/src/main/java/org/apache/flume/source/SpoolDirectorySourceConfigurationConstants.java 806a661
  flume-ng-core/src/main/java/org/apache/flume/tools/PlatformDetect.java PRE-CREATION
  flume-ng-core/src/test/java/org/apache/flume/client/avro/TestSpoolingFileLineReader.java 740bc98
  flume-ng-core/src/test/java/org/apache/flume/serialization/ResettableTestStringInputStream.java PRE-CREATION
  flume-ng-core/src/test/java/org/apache/flume/serialization/TestDurablePositionTracker.java PRE-CREATION
  flume-ng-core/src/test/java/org/apache/flume/serialization/TestLineDeserializer.java PRE-CREATION
  flume-ng-core/src/test/java/org/apache/flume/serialization/TestResettableFileInputStream.java PRE-CREATION
  flume-ng-core/src/test/java/org/apache/flume/serialization/TestSpoolingFileEventParser.java PRE-CREATION
  flume-ng-core/src/test/java/org/apache/flume/source/TestSpoolDirectorySource.java 6e87b21
  flume-ng-core/src/test/resources/TestResettableFileInputStream_1.avro PRE-CREATION
  flume-ng-core/src/test/resources/TestResettableFileInputStream_1.truncated.avro PRE-CREATION
  pom.xml b934c1d

Diff: https://reviews.apache.org/r/8596/diff/
Testing (updated)

Unit tests pass.
Thanks,

Mike Percy

Re: Review Request: FLUME-1632: Persist progress on each file in file spooling client/source

Mike Percy 2012-12-17, 08:58

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/8596/
-----------------------------------------------------------

(Updated Dec. 17, 2012, 8:58 a.m.)
Review request for Flume.
Changes
-------

Cleaned up this patch a lot. This is ready for review. I removed the forwarder stuff for now and implemented support for the avro-client.
Description
-------

Defines EventDeserializer interface and uses it from the spooling source. Progress is persisted as bytes are read from the underlying file.
This addresses bug FLUME-1632.
   https://issues.apache.org/jira/browse/FLUME-1632
Diffs (updated)
-----

  flume-ng-core/pom.xml 0224519
  flume-ng-core/src/main/avro/TransferStateFileMeta.avsc PRE-CREATION
  flume-ng-core/src/main/java/org/apache/flume/client/avro/AvroCLIClient.java 37e9ffa
  flume-ng-core/src/main/java/org/apache/flume/client/avro/BufferedLineReader.java 718e1b2
  flume-ng-core/src/main/java/org/apache/flume/client/avro/EventReader.java PRE-CREATION
  flume-ng-core/src/main/java/org/apache/flume/client/avro/LineReader.java 904f22c
  flume-ng-core/src/main/java/org/apache/flume/client/avro/ReliableEventReader.java PRE-CREATION
  flume-ng-core/src/main/java/org/apache/flume/client/avro/ReliableSpoolingFileEventReader.java PRE-CREATION
  flume-ng-core/src/main/java/org/apache/flume/client/avro/SimpleTextLineEventReader.java PRE-CREATION
  flume-ng-core/src/main/java/org/apache/flume/client/avro/SpoolingFileLineReader.java 8362299
  flume-ng-core/src/main/java/org/apache/flume/serialization/DurablePositionTracker.java PRE-CREATION
  flume-ng-core/src/main/java/org/apache/flume/serialization/EventDeserializer.java PRE-CREATION
  flume-ng-core/src/main/java/org/apache/flume/serialization/EventDeserializerFactory.java PRE-CREATION
  flume-ng-core/src/main/java/org/apache/flume/serialization/EventDeserializerType.java PRE-CREATION
  flume-ng-core/src/main/java/org/apache/flume/serialization/EventSerDe.java PRE-CREATION
  flume-ng-core/src/main/java/org/apache/flume/serialization/EventSerializer.java a418935
  flume-ng-core/src/main/java/org/apache/flume/serialization/EventSerializerFactory.java 75853a9
  flume-ng-core/src/main/java/org/apache/flume/serialization/EventSerializerType.java afe8ed8
  flume-ng-core/src/main/java/org/apache/flume/serialization/LineDeserializer.java PRE-CREATION
  flume-ng-core/src/main/java/org/apache/flume/serialization/PositionTracker.java PRE-CREATION
  flume-ng-core/src/main/java/org/apache/flume/serialization/Resettable.java PRE-CREATION
  flume-ng-core/src/main/java/org/apache/flume/serialization/ResettableFileInputStream.java PRE-CREATION
  flume-ng-core/src/main/java/org/apache/flume/serialization/ResettableInputStream.java PRE-CREATION
  flume-ng-core/src/main/java/org/apache/flume/source/SpoolDirectorySource.java 61824d8
  flume-ng-core/src/main/java/org/apache/flume/source/SpoolDirectorySourceConfigurationConstants.java 806a661
  flume-ng-core/src/main/java/org/apache/flume/tools/PlatformDetect.java PRE-CREATION
  flume-ng-core/src/test/java/org/apache/flume/client/avro/TestBufferedLineReader.java 169abe5
  flume-ng-core/src/test/java/org/apache/flume/client/avro/TestSpoolingFileLineReader.java 740bc98
  flume-ng-core/src/test/java/org/apache/flume/serialization/ResettableTestStringInputStream.java PRE-CREATION
  flume-ng-core/src/test/java/org/apache/flume/serialization/TestDurablePositionTracker.java PRE-CREATION
  flume-ng-core/src/test/java/org/apache/flume/serialization/TestLineDeserializer.java PRE-CREATION
  flume-ng-core/src/test/java/org/apache/flume/serialization/TestResettableFileInputStream.java PRE-CREATION
  flume-ng-core/src/test/java/org/apache/flume/serialization/TestSpoolingFileEventParser.java PRE-CREATION
  flume-ng-core/src/test/java/org/apache/flume/source/TestSpoolDirectorySource.java 6e87b21
  flume-ng-core/src/test/resources/TestResettableFileInputStream_1.avro PRE-CREATION
  flume-ng-core/src/test/resources/TestResettableFileInputStream_1.truncated.avro PRE-CREATION
  pom.xml b934c1d

Diff: https://reviews.apache.org/r/8596/diff/
Testing

Unit tests pass.
Thanks,

Mike Percy

Re: Review Request: FLUME-1632: Persist progress on each file in file spooling client/source

Brock Noland 2012-12-18, 17:44

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/8596/#review14583
-----------------------------------------------------------
Looks good, have you done any manual testing of the change?
flume-ng-core/src/main/java/org/apache/flume/client/avro/ReliableSpoolingFileEventReader.java
<https://reviews.apache.org/r/8596/#comment31014>

   Extra space in error message.

flume-ng-core/src/main/java/org/apache/flume/serialization/ResettableFileInputStream.java
<https://reviews.apache.org/r/8596/#comment31016>

   Why use direct memory here?

flume-ng-core/src/main/java/org/apache/flume/serialization/ResettableFileInputStream.java
<https://reviews.apache.org/r/8596/#comment31015>

   What is the logic here, why 4?
- Brock Noland
On Dec. 17, 2012, 8:58 a.m., Mike Percy wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/8596/
> -----------------------------------------------------------
>
> (Updated Dec. 17, 2012, 8:58 a.m.)
>
>
> Review request for Flume.
>
>
> Description
> -------
>
> Defines EventDeserializer interface and uses it from the spooling source. Progress is persisted as bytes are read from the underlying file.
>
>
> This addresses bug FLUME-1632.
> https://issues.apache.org/jira/browse/FLUME-1632
>
>
> Diffs
> -----
>
> flume-ng-core/pom.xml 0224519
> flume-ng-core/src/main/avro/TransferStateFileMeta.avsc PRE-CREATION
> flume-ng-core/src/main/java/org/apache/flume/client/avro/AvroCLIClient.java 37e9ffa
> flume-ng-core/src/main/java/org/apache/flume/client/avro/BufferedLineReader.java 718e1b2
> flume-ng-core/src/main/java/org/apache/flume/client/avro/EventReader.java PRE-CREATION
> flume-ng-core/src/main/java/org/apache/flume/client/avro/LineReader.java 904f22c
> flume-ng-core/src/main/java/org/apache/flume/client/avro/ReliableEventReader.java PRE-CREATION
> flume-ng-core/src/main/java/org/apache/flume/client/avro/ReliableSpoolingFileEventReader.java PRE-CREATION
> flume-ng-core/src/main/java/org/apache/flume/client/avro/SimpleTextLineEventReader.java PRE-CREATION
> flume-ng-core/src/main/java/org/apache/flume/client/avro/SpoolingFileLineReader.java 8362299
> flume-ng-core/src/main/java/org/apache/flume/serialization/DurablePositionTracker.java PRE-CREATION
> flume-ng-core/src/main/java/org/apache/flume/serialization/EventDeserializer.java PRE-CREATION
> flume-ng-core/src/main/java/org/apache/flume/serialization/EventDeserializerFactory.java PRE-CREATION
> flume-ng-core/src/main/java/org/apache/flume/serialization/EventDeserializerType.java PRE-CREATION
> flume-ng-core/src/main/java/org/apache/flume/serialization/EventSerDe.java PRE-CREATION
> flume-ng-core/src/main/java/org/apache/flume/serialization/EventSerializer.java a418935
> flume-ng-core/src/main/java/org/apache/flume/serialization/EventSerializerFactory.java 75853a9
> flume-ng-core/src/main/java/org/apache/flume/serialization/EventSerializerType.java afe8ed8
> flume-ng-core/src/main/java/org/apache/flume/serialization/LineDeserializer.java PRE-CREATION
> flume-ng-core/src/main/java/org/apache/flume/serialization/PositionTracker.java PRE-CREATION
> flume-ng-core/src/main/java/org/apache/flume/serialization/Resettable.java PRE-CREATION
> flume-ng-core/src/main/java/org/apache/flume/serialization/ResettableFileInputStream.java PRE-CREATION
> flume-ng-core/src/main/java/org/apache/flume/serialization/ResettableInputStream.java PRE-CREATION
> flume-ng-core/src/main/java/org/apache/flume/source/SpoolDirectorySource.java 61824d8
> flume-ng-core/src/main/java/org/apache/flume/source/SpoolDirectorySourceConfigurationConstants.java 806a661
> flume-ng-core/src/main/java/org/apache/flume/tools/PlatformDetect.java PRE-CREATION
> flume-ng-core/src/test/java/org/apache/flume/client/avro/TestBufferedLineReader.java 169abe5

Re: Review Request: FLUME-1632: Persist progress on each file in file spooling client/source

Mike Percy 2012-12-18, 21:25

> On Dec. 18, 2012, 5:44 p.m., Brock Noland wrote:
> > Looks good, have you done any manual testing of the change?

I think I found an issue with the source implementation actually, digging into it now. It processes the files but I'm seeing an issue once the directory is empty that I'm having trouble reproducing with unit tests.
> On Dec. 18, 2012, 5:44 p.m., Brock Noland wrote:
> > flume-ng-core/src/main/java/org/apache/flume/serialization/ResettableFileInputStream.java, line 98
> > <https://reviews.apache.org/r/8596/diff/3/?file=239304#file239304line98>
> >
> > Why use direct memory here?

At least in OpenJDK, when reading from a file if you use a heap buffer it internally allocates a direct buffer then copies to the heap buffer. This is supposed to be a performance optimization.
> On Dec. 18, 2012, 5:44 p.m., Brock Noland wrote:
> > flume-ng-core/src/main/java/org/apache/flume/serialization/ResettableFileInputStream.java, line 123
> > <https://reviews.apache.org/r/8596/diff/3/?file=239304#file239304line123>
> >
> > What is the logic here, why 4?

32 bits is the maximum size that a single character should ever take up in Unicode.
- Mike
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/8596/#review14583
-----------------------------------------------------------
On Dec. 17, 2012, 8:58 a.m., Mike Percy wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/8596/
> -----------------------------------------------------------
>
> (Updated Dec. 17, 2012, 8:58 a.m.)
>
>
> Review request for Flume.
>
>
> Description
> -------
>
> Defines EventDeserializer interface and uses it from the spooling source. Progress is persisted as bytes are read from the underlying file.
>
>
> This addresses bug FLUME-1632.
> https://issues.apache.org/jira/browse/FLUME-1632
>
>
> Diffs
> -----
>
> flume-ng-core/pom.xml 0224519
> flume-ng-core/src/main/avro/TransferStateFileMeta.avsc PRE-CREATION
> flume-ng-core/src/main/java/org/apache/flume/client/avro/AvroCLIClient.java 37e9ffa
> flume-ng-core/src/main/java/org/apache/flume/client/avro/BufferedLineReader.java 718e1b2
> flume-ng-core/src/main/java/org/apache/flume/client/avro/EventReader.java PRE-CREATION
> flume-ng-core/src/main/java/org/apache/flume/client/avro/LineReader.java 904f22c
> flume-ng-core/src/main/java/org/apache/flume/client/avro/ReliableEventReader.java PRE-CREATION
> flume-ng-core/src/main/java/org/apache/flume/client/avro/ReliableSpoolingFileEventReader.java PRE-CREATION
> flume-ng-core/src/main/java/org/apache/flume/client/avro/SimpleTextLineEventReader.java PRE-CREATION
> flume-ng-core/src/main/java/org/apache/flume/client/avro/SpoolingFileLineReader.java 8362299
> flume-ng-core/src/main/java/org/apache/flume/serialization/DurablePositionTracker.java PRE-CREATION
> flume-ng-core/src/main/java/org/apache/flume/serialization/EventDeserializer.java PRE-CREATION
> flume-ng-core/src/main/java/org/apache/flume/serialization/EventDeserializerFactory.java PRE-CREATION
> flume-ng-core/src/main/java/org/apache/flume/serialization/EventDeserializerType.java PRE-CREATION
> flume-ng-core/src/main/java/org/apache/flume/serialization/EventSerDe.java PRE-CREATION
> flume-ng-core/src/main/java/org/apache/flume/serialization/EventSerializer.java a418935
> flume-ng-core/src/main/java/org/apache/flume/serialization/EventSerializerFactory.java 75853a9
> flume-ng-core/src/main/java/org/apache/flume/serialization/EventSerializerType.java afe8ed8
> flume-ng-core/src/main/java/org/apache/flume/serialization/LineDeserializer.java PRE-CREATION
> flume-ng-core/src/main/java/org/apache/flume/serialization/PositionTracker.java PRE-CREATION
> flume-ng-core/src/main/java/org/apache/flume/serialization/Resettable.java PRE-CREATION

Re: Review Request: FLUME-1632: Persist progress on each file in file spooling client/source

Brock Noland 2012-12-18, 21:29

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/8596/#review14686
-----------------------------------------------------------

flume-ng-core/src/main/java/org/apache/flume/serialization/ResettableFileInputStream.java
<https://reviews.apache.org/r/8596/#comment31148>

I figured it was something like that but I think that section could use a few comments describing what is going on.
- Brock Noland
On Dec. 17, 2012, 8:58 a.m., Mike Percy wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/8596/
> -----------------------------------------------------------
>
> (Updated Dec. 17, 2012, 8:58 a.m.)
>
>
> Review request for Flume.
>
>
> Description
> -------
>
> Defines EventDeserializer interface and uses it from the spooling source. Progress is persisted as bytes are read from the underlying file.
>
>
> This addresses bug FLUME-1632.
> https://issues.apache.org/jira/browse/FLUME-1632
>
>
> Diffs
> -----
>
> flume-ng-core/pom.xml 0224519
> flume-ng-core/src/main/avro/TransferStateFileMeta.avsc PRE-CREATION
> flume-ng-core/src/main/java/org/apache/flume/client/avro/AvroCLIClient.java 37e9ffa
> flume-ng-core/src/main/java/org/apache/flume/client/avro/BufferedLineReader.java 718e1b2
> flume-ng-core/src/main/java/org/apache/flume/client/avro/EventReader.java PRE-CREATION
> flume-ng-core/src/main/java/org/apache/flume/client/avro/LineReader.java 904f22c
> flume-ng-core/src/main/java/org/apache/flume/client/avro/ReliableEventReader.java PRE-CREATION
> flume-ng-core/src/main/java/org/apache/flume/client/avro/ReliableSpoolingFileEventReader.java PRE-CREATION
> flume-ng-core/src/main/java/org/apache/flume/client/avro/SimpleTextLineEventReader.java PRE-CREATION
> flume-ng-core/src/main/java/org/apache/flume/client/avro/SpoolingFileLineReader.java 8362299
> flume-ng-core/src/main/java/org/apache/flume/serialization/DurablePositionTracker.java PRE-CREATION
> flume-ng-core/src/main/java/org/apache/flume/serialization/EventDeserializer.java PRE-CREATION
> flume-ng-core/src/main/java/org/apache/flume/serialization/EventDeserializerFactory.java PRE-CREATION
> flume-ng-core/src/main/java/org/apache/flume/serialization/EventDeserializerType.java PRE-CREATION
> flume-ng-core/src/main/java/org/apache/flume/serialization/EventSerDe.java PRE-CREATION
> flume-ng-core/src/main/java/org/apache/flume/serialization/EventSerializer.java a418935
> flume-ng-core/src/main/java/org/apache/flume/serialization/EventSerializerFactory.java 75853a9
> flume-ng-core/src/main/java/org/apache/flume/serialization/EventSerializerType.java afe8ed8
> flume-ng-core/src/main/java/org/apache/flume/serialization/LineDeserializer.java PRE-CREATION
> flume-ng-core/src/main/java/org/apache/flume/serialization/PositionTracker.java PRE-CREATION
> flume-ng-core/src/main/java/org/apache/flume/serialization/Resettable.java PRE-CREATION
> flume-ng-core/src/main/java/org/apache/flume/serialization/ResettableFileInputStream.java PRE-CREATION
> flume-ng-core/src/main/java/org/apache/flume/serialization/ResettableInputStream.java PRE-CREATION
> flume-ng-core/src/main/java/org/apache/flume/source/SpoolDirectorySource.java 61824d8
> flume-ng-core/src/main/java/org/apache/flume/source/SpoolDirectorySourceConfigurationConstants.java 806a661
> flume-ng-core/src/main/java/org/apache/flume/tools/PlatformDetect.java PRE-CREATION
> flume-ng-core/src/test/java/org/apache/flume/client/avro/TestBufferedLineReader.java 169abe5
> flume-ng-core/src/test/java/org/apache/flume/client/avro/TestSpoolingFileLineReader.java 740bc98
> flume-ng-core/src/test/java/org/apache/flume/serialization/ResettableTestStringInputStream.java PRE-CREATION
> flume-ng-core/src/test/java/org/apache/flume/serialization/TestDurablePositionTracker.java PRE-CREATION

Re: Review Request: FLUME-1632: Persist progress on each file in file spooling client/source

Mike Percy 2012-12-18, 21:50

> On Dec. 18, 2012, 9:29 p.m., Brock Noland wrote:
> > flume-ng-core/src/main/java/org/apache/flume/serialization/ResettableFileInputStream.java, line 123
> > <https://reviews.apache.org/r/8596/diff/3/?file=239304#file239304line123>
> >
> > I figured it was something like that but I think that section could use a few comments describing what is going on.

You're right... and you successfully homed in on the trickiest / jankiest part of this patch... :) I will improve the docs for that part.
- Mike
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/8596/#review14686
-----------------------------------------------------------
On Dec. 17, 2012, 8:58 a.m., Mike Percy wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/8596/
> -----------------------------------------------------------
>
> (Updated Dec. 17, 2012, 8:58 a.m.)
>
>
> Review request for Flume.
>
>
> Description
> -------
>
> Defines EventDeserializer interface and uses it from the spooling source. Progress is persisted as bytes are read from the underlying file.
>
>
> This addresses bug FLUME-1632.
> https://issues.apache.org/jira/browse/FLUME-1632
>
>
> Diffs
> -----
>
> flume-ng-core/pom.xml 0224519
> flume-ng-core/src/main/avro/TransferStateFileMeta.avsc PRE-CREATION
> flume-ng-core/src/main/java/org/apache/flume/client/avro/AvroCLIClient.java 37e9ffa
> flume-ng-core/src/main/java/org/apache/flume/client/avro/BufferedLineReader.java 718e1b2
> flume-ng-core/src/main/java/org/apache/flume/client/avro/EventReader.java PRE-CREATION
> flume-ng-core/src/main/java/org/apache/flume/client/avro/LineReader.java 904f22c
> flume-ng-core/src/main/java/org/apache/flume/client/avro/ReliableEventReader.java PRE-CREATION
> flume-ng-core/src/main/java/org/apache/flume/client/avro/ReliableSpoolingFileEventReader.java PRE-CREATION
> flume-ng-core/src/main/java/org/apache/flume/client/avro/SimpleTextLineEventReader.java PRE-CREATION
> flume-ng-core/src/main/java/org/apache/flume/client/avro/SpoolingFileLineReader.java 8362299
> flume-ng-core/src/main/java/org/apache/flume/serialization/DurablePositionTracker.java PRE-CREATION
> flume-ng-core/src/main/java/org/apache/flume/serialization/EventDeserializer.java PRE-CREATION
> flume-ng-core/src/main/java/org/apache/flume/serialization/EventDeserializerFactory.java PRE-CREATION
> flume-ng-core/src/main/java/org/apache/flume/serialization/EventDeserializerType.java PRE-CREATION
> flume-ng-core/src/main/java/org/apache/flume/serialization/EventSerDe.java PRE-CREATION
> flume-ng-core/src/main/java/org/apache/flume/serialization/EventSerializer.java a418935
> flume-ng-core/src/main/java/org/apache/flume/serialization/EventSerializerFactory.java 75853a9
> flume-ng-core/src/main/java/org/apache/flume/serialization/EventSerializerType.java afe8ed8
> flume-ng-core/src/main/java/org/apache/flume/serialization/LineDeserializer.java PRE-CREATION
> flume-ng-core/src/main/java/org/apache/flume/serialization/PositionTracker.java PRE-CREATION
> flume-ng-core/src/main/java/org/apache/flume/serialization/Resettable.java PRE-CREATION
> flume-ng-core/src/main/java/org/apache/flume/serialization/ResettableFileInputStream.java PRE-CREATION
> flume-ng-core/src/main/java/org/apache/flume/serialization/ResettableInputStream.java PRE-CREATION
> flume-ng-core/src/main/java/org/apache/flume/source/SpoolDirectorySource.java 61824d8
> flume-ng-core/src/main/java/org/apache/flume/source/SpoolDirectorySourceConfigurationConstants.java 806a661
> flume-ng-core/src/main/java/org/apache/flume/tools/PlatformDetect.java PRE-CREATION
> flume-ng-core/src/test/java/org/apache/flume/client/avro/TestBufferedLineReader.java 169abe5
> flume-ng-core/src/test/java/org/apache/flume/client/avro/TestSpoolingFileLineReader.java 740bc98

Re: Review Request: FLUME-1632: Persist progress on each file in file spooling client/source

Mike Percy 2012-12-18, 23:22

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/8596/
-----------------------------------------------------------

(Updated Dec. 18, 2012, 11:21 p.m.)
Review request for Flume.
Changes
-------

Fixed an issue with empty line handling and cleaned up the byte reading code.
Description
-------

Defines EventDeserializer interface and uses it from the spooling source. Progress is persisted as bytes are read from the underlying file.
This addresses bug FLUME-1632.
   https://issues.apache.org/jira/browse/FLUME-1632
Diffs (updated)
-----

  flume-ng-core/pom.xml 0224519
  flume-ng-core/src/main/avro/TransferStateFileMeta.avsc PRE-CREATION
  flume-ng-core/src/main/java/org/apache/flume/client/avro/AvroCLIClient.java 37e9ffa
  flume-ng-core/src/main/java/org/apache/flume/client/avro/BufferedLineReader.java 718e1b2
  flume-ng-core/src/main/java/org/apache/flume/client/avro/EventReader.java PRE-CREATION
  flume-ng-core/src/main/java/org/apache/flume/client/avro/LineReader.java 904f22c
  flume-ng-core/src/main/java/org/apache/flume/client/avro/ReliableEventReader.java PRE-CREATION
  flume-ng-core/src/main/java/org/apache/flume/client/avro/ReliableSpoolingFileEventReader.java PRE-CREATION
  flume-ng-core/src/main/java/org/apache/flume/client/avro/SimpleTextLineEventReader.java PRE-CREATION
  flume-ng-core/src/main/java/org/apache/flume/client/avro/SpoolingFileLineReader.java 8362299
  flume-ng-core/src/main/java/org/apache/flume/serialization/DurablePositionTracker.java PRE-CREATION
  flume-ng-core/src/main/java/org/apache/flume/serialization/EventDeserializer.java PRE-CREATION
  flume-ng-core/src/main/java/org/apache/flume/serialization/EventDeserializerFactory.java PRE-CREATION
  flume-ng-core/src/main/java/org/apache/flume/serialization/EventDeserializerType.java PRE-CREATION
  flume-ng-core/src/main/java/org/apache/flume/serialization/EventSerDe.java PRE-CREATION
  flume-ng-core/src/main/java/org/apache/flume/serialization/EventSerializer.java a418935
  flume-ng-core/src/main/java/org/apache/flume/serialization/EventSerializerFactory.java 75853a9
  flume-ng-core/src/main/java/org/apache/flume/serialization/EventSerializerType.java afe8ed8
  flume-ng-core/src/main/java/org/apache/flume/serialization/LineDeserializer.java PRE-CREATION
  flume-ng-core/src/main/java/org/apache/flume/serialization/PositionTracker.java PRE-CREATION
  flume-ng-core/src/main/java/org/apache/flume/serialization/Resettable.java PRE-CREATION
  flume-ng-core/src/main/java/org/apache/flume/serialization/ResettableFileInputStream.java PRE-CREATION
  flume-ng-core/src/main/java/org/apache/flume/serialization/ResettableInputStream.java PRE-CREATION
  flume-ng-core/src/main/java/org/apache/flume/source/SpoolDirectorySource.java 61824d8
  flume-ng-core/src/main/java/org/apache/flume/source/SpoolDirectorySourceConfigurationConstants.java 806a661
  flume-ng-core/src/main/java/org/apache/flume/tools/PlatformDetect.java PRE-CREATION
  flume-ng-core/src/test/java/org/apache/flume/client/avro/TestBufferedLineReader.java 169abe5
  flume-ng-core/src/test/java/org/apache/flume/client/avro/TestSpoolingFileLineReader.java 740bc98
  flume-ng-core/src/test/java/org/apache/flume/serialization/ResettableTestStringInputStream.java PRE-CREATION
  flume-ng-core/src/test/java/org/apache/flume/serialization/TestDurablePositionTracker.java PRE-CREATION
  flume-ng-core/src/test/java/org/apache/flume/serialization/TestLineDeserializer.java PRE-CREATION
  flume-ng-core/src/test/java/org/apache/flume/serialization/TestReliableSpoolingFileEventReader.java PRE-CREATION
  flume-ng-core/src/test/java/org/apache/flume/serialization/TestResettableFileInputStream.java PRE-CREATION
  flume-ng-core/src/test/java/org/apache/flume/source/TestSpoolDirectorySource.java 6e87b21
  flume-ng-core/src/test/resources/TestResettableFileInputStream_1.avro PRE-CREATION
  flume-ng-core/src/test/resources/TestResettableFileInputStream_1.truncated.avro PRE-CREATION
  pom.xml b934c1d

Diff: https://reviews.apache.org/r/8596/diff/
Testing

Unit tests pass.
Thanks,

Mike Percy

Re: Review Request: FLUME-1632: Persist progress on each file in file spooling client/source

Brock Noland 2012-12-19, 00:00

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/8596/#review14696
-----------------------------------------------------------

flume-ng-core/src/main/java/org/apache/flume/serialization/ResettableFileInputStream.java
<https://reviews.apache.org/r/8596/#comment31158>

Nevermind, I thought we were calling this method http://docs.oracle.com/javase/1.4.2/docs/api/java/io/InputStream.html#read() which we are not.
- Brock Noland
On Dec. 18, 2012, 11:21 p.m., Mike Percy wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/8596/
> -----------------------------------------------------------
>
> (Updated Dec. 18, 2012, 11:21 p.m.)
>
>
> Review request for Flume.
>
>
> Description
> -------
>
> Defines EventDeserializer interface and uses it from the spooling source. Progress is persisted as bytes are read from the underlying file.
>
>
> This addresses bug FLUME-1632.
> https://issues.apache.org/jira/browse/FLUME-1632
>
>
> Diffs
> -----
>
> flume-ng-core/pom.xml 0224519
> flume-ng-core/src/main/avro/TransferStateFileMeta.avsc PRE-CREATION
> flume-ng-core/src/main/java/org/apache/flume/client/avro/AvroCLIClient.java 37e9ffa
> flume-ng-core/src/main/java/org/apache/flume/client/avro/BufferedLineReader.java 718e1b2
> flume-ng-core/src/main/java/org/apache/flume/client/avro/EventReader.java PRE-CREATION
> flume-ng-core/src/main/java/org/apache/flume/client/avro/LineReader.java 904f22c
> flume-ng-core/src/main/java/org/apache/flume/client/avro/ReliableEventReader.java PRE-CREATION
> flume-ng-core/src/main/java/org/apache/flume/client/avro/ReliableSpoolingFileEventReader.java PRE-CREATION
> flume-ng-core/src/main/java/org/apache/flume/client/avro/SimpleTextLineEventReader.java PRE-CREATION
> flume-ng-core/src/main/java/org/apache/flume/client/avro/SpoolingFileLineReader.java 8362299
> flume-ng-core/src/main/java/org/apache/flume/serialization/DurablePositionTracker.java PRE-CREATION
> flume-ng-core/src/main/java/org/apache/flume/serialization/EventDeserializer.java PRE-CREATION
> flume-ng-core/src/main/java/org/apache/flume/serialization/EventDeserializerFactory.java PRE-CREATION
> flume-ng-core/src/main/java/org/apache/flume/serialization/EventDeserializerType.java PRE-CREATION
> flume-ng-core/src/main/java/org/apache/flume/serialization/EventSerDe.java PRE-CREATION
> flume-ng-core/src/main/java/org/apache/flume/serialization/EventSerializer.java a418935
> flume-ng-core/src/main/java/org/apache/flume/serialization/EventSerializerFactory.java 75853a9
> flume-ng-core/src/main/java/org/apache/flume/serialization/EventSerializerType.java afe8ed8
> flume-ng-core/src/main/java/org/apache/flume/serialization/LineDeserializer.java PRE-CREATION
> flume-ng-core/src/main/java/org/apache/flume/serialization/PositionTracker.java PRE-CREATION
> flume-ng-core/src/main/java/org/apache/flume/serialization/Resettable.java PRE-CREATION
> flume-ng-core/src/main/java/org/apache/flume/serialization/ResettableFileInputStream.java PRE-CREATION
> flume-ng-core/src/main/java/org/apache/flume/serialization/ResettableInputStream.java PRE-CREATION
> flume-ng-core/src/main/java/org/apache/flume/source/SpoolDirectorySource.java 61824d8
> flume-ng-core/src/main/java/org/apache/flume/source/SpoolDirectorySourceConfigurationConstants.java 806a661
> flume-ng-core/src/main/java/org/apache/flume/tools/PlatformDetect.java PRE-CREATION
> flume-ng-core/src/test/java/org/apache/flume/client/avro/TestBufferedLineReader.java 169abe5
> flume-ng-core/src/test/java/org/apache/flume/client/avro/TestSpoolingFileLineReader.java 740bc98
> flume-ng-core/src/test/java/org/apache/flume/serialization/ResettableTestStringInputStream.java PRE-CREATION
> flume-ng-core/src/test/java/org/apache/flume/serialization/TestDurablePositionTracker.java PRE-CREATION

Re: Review Request: FLUME-1632: Persist progress on each file in file spooling client/source

Brock Noland 2012-12-18, 23:42

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/8596/#review14694
-----------------------------------------------------------

flume-ng-core/src/main/java/org/apache/flume/serialization/ResettableFileInputStream.java
<https://reviews.apache.org/r/8596/#comment31157>

Isn't 0 (Null) a valid byte? If we are seeing this, perhaps we are seeking past the end of the file and creating a sparse file?
- Brock Noland
On Dec. 18, 2012, 11:21 p.m., Mike Percy wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/8596/
> -----------------------------------------------------------
>
> (Updated Dec. 18, 2012, 11:21 p.m.)
>
>
> Review request for Flume.
>
>
> Description
> -------
>
> Defines EventDeserializer interface and uses it from the spooling source. Progress is persisted as bytes are read from the underlying file.
>
>
> This addresses bug FLUME-1632.
> https://issues.apache.org/jira/browse/FLUME-1632
>
>
> Diffs
> -----
>
> flume-ng-core/pom.xml 0224519
> flume-ng-core/src/main/avro/TransferStateFileMeta.avsc PRE-CREATION
> flume-ng-core/src/main/java/org/apache/flume/client/avro/AvroCLIClient.java 37e9ffa
> flume-ng-core/src/main/java/org/apache/flume/client/avro/BufferedLineReader.java 718e1b2
> flume-ng-core/src/main/java/org/apache/flume/client/avro/EventReader.java PRE-CREATION
> flume-ng-core/src/main/java/org/apache/flume/client/avro/LineReader.java 904f22c
> flume-ng-core/src/main/java/org/apache/flume/client/avro/ReliableEventReader.java PRE-CREATION
> flume-ng-core/src/main/java/org/apache/flume/client/avro/ReliableSpoolingFileEventReader.java PRE-CREATION
> flume-ng-core/src/main/java/org/apache/flume/client/avro/SimpleTextLineEventReader.java PRE-CREATION
> flume-ng-core/src/main/java/org/apache/flume/client/avro/SpoolingFileLineReader.java 8362299
> flume-ng-core/src/main/java/org/apache/flume/serialization/DurablePositionTracker.java PRE-CREATION
> flume-ng-core/src/main/java/org/apache/flume/serialization/EventDeserializer.java PRE-CREATION
> flume-ng-core/src/main/java/org/apache/flume/serialization/EventDeserializerFactory.java PRE-CREATION
> flume-ng-core/src/main/java/org/apache/flume/serialization/EventDeserializerType.java PRE-CREATION
> flume-ng-core/src/main/java/org/apache/flume/serialization/EventSerDe.java PRE-CREATION
> flume-ng-core/src/main/java/org/apache/flume/serialization/EventSerializer.java a418935
> flume-ng-core/src/main/java/org/apache/flume/serialization/EventSerializerFactory.java 75853a9
> flume-ng-core/src/main/java/org/apache/flume/serialization/EventSerializerType.java afe8ed8
> flume-ng-core/src/main/java/org/apache/flume/serialization/LineDeserializer.java PRE-CREATION
> flume-ng-core/src/main/java/org/apache/flume/serialization/PositionTracker.java PRE-CREATION
> flume-ng-core/src/main/java/org/apache/flume/serialization/Resettable.java PRE-CREATION
> flume-ng-core/src/main/java/org/apache/flume/serialization/ResettableFileInputStream.java PRE-CREATION
> flume-ng-core/src/main/java/org/apache/flume/serialization/ResettableInputStream.java PRE-CREATION
> flume-ng-core/src/main/java/org/apache/flume/source/SpoolDirectorySource.java 61824d8
> flume-ng-core/src/main/java/org/apache/flume/source/SpoolDirectorySourceConfigurationConstants.java 806a661
> flume-ng-core/src/main/java/org/apache/flume/tools/PlatformDetect.java PRE-CREATION
> flume-ng-core/src/test/java/org/apache/flume/client/avro/TestBufferedLineReader.java 169abe5
> flume-ng-core/src/test/java/org/apache/flume/client/avro/TestSpoolingFileLineReader.java 740bc98
> flume-ng-core/src/test/java/org/apache/flume/serialization/ResettableTestStringInputStream.java PRE-CREATION
> flume-ng-core/src/test/java/org/apache/flume/serialization/TestDurablePositionTracker.java PRE-CREATION

Re: Review Request: FLUME-1632: Persist progress on each file in file spooling client/source

Brock Noland 2012-12-19, 01:54

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/8596/#review14708
-----------------------------------------------------------
Hi Mike!

Great patch! There are a couple issues below but otherwise I think it's ready for a commit!
flume-ng-core/src/main/java/org/apache/flume/client/avro/AvroCLIClient.java
<https://reviews.apache.org/r/8596/#comment31192>

   It's quite slow with such a large batch size. 4 minutes to transfer 60MB of data to a local agent (memory channel and null sink). It would be nice if that was configurable. That could be a follow up JIRA but it'd be nice to set this be a command line option.

flume-ng-core/src/main/java/org/apache/flume/client/avro/AvroCLIClient.java
<https://reviews.apache.org/r/8596/#comment31190>

   I know you didn't change this, but I hit this while testing. That exception should be logged in the logger.error().

flume-ng-core/src/main/java/org/apache/flume/client/avro/ReliableSpoolingFileEventReader.java
<https://reviews.apache.org/r/8596/#comment31189>

   It's possible for currentFile to be absent.

flume-ng-core/src/main/java/org/apache/flume/serialization/LineDeserializer.java
<https://reviews.apache.org/r/8596/#comment31191>

   Seeing the error below. Guessing it's cause the last file during a run will be closed twice. One in retireCurrentFile and once in the close method. This was doubly ugly for me because it hid an exception being thrown in retireCurrentFile due to a file name violation.


   java.nio.channels.ClosedChannelException
   at sun.nio.ch.FileChannelImpl.ensureOpen(FileChannelImpl.java:88)
   at sun.nio.ch.FileChannelImpl.position(FileChannelImpl.java:265)
   at org.apache.flume.serialization.ResettableFileInputStream.seek(ResettableFileInputStream.java:212)
   at org.apache.flume.serialization.ResettableFileInputStream.reset(ResettableFileInputStream.java:204)
   at org.apache.flume.serialization.LineDeserializer.reset(LineDeserializer.java:102)
   at org.apache.flume.serialization.LineDeserializer.close(LineDeserializer.java:107)
   at org.apache.flume.client.avro.ReliableSpoolingFileEventReader.close(ReliableSpoolingFileEventReader.java:224)
   at org.apache.flume.client.avro.AvroCLIClient.run(AvroCLIClient.java:217)
   at org.apache.flume.client.avro.AvroCLIClient.main(AvroCLIClient.java:71)


- Brock Noland
On Dec. 18, 2012, 11:21 p.m., Mike Percy wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/8596/
> -----------------------------------------------------------
>
> (Updated Dec. 18, 2012, 11:21 p.m.)
>
>
> Review request for Flume.
>
>
> Description
> -------
>
> Defines EventDeserializer interface and uses it from the spooling source. Progress is persisted as bytes are read from the underlying file.
>
>
> This addresses bug FLUME-1632.
> https://issues.apache.org/jira/browse/FLUME-1632
>
>
> Diffs
> -----
>
> flume-ng-core/pom.xml 0224519
> flume-ng-core/src/main/avro/TransferStateFileMeta.avsc PRE-CREATION
> flume-ng-core/src/main/java/org/apache/flume/client/avro/AvroCLIClient.java 37e9ffa
> flume-ng-core/src/main/java/org/apache/flume/client/avro/BufferedLineReader.java 718e1b2
> flume-ng-core/src/main/java/org/apache/flume/client/avro/EventReader.java PRE-CREATION
> flume-ng-core/src/main/java/org/apache/flume/client/avro/LineReader.java 904f22c
> flume-ng-core/src/main/java/org/apache/flume/client/avro/ReliableEventReader.java PRE-CREATION
> flume-ng-core/src/main/java/org/apache/flume/client/avro/ReliableSpoolingFileEventReader.java PRE-CREATION
> flume-ng-core/src/main/java/org/apache/flume/client/avro/SimpleTextLineEventReader.java PRE-CREATION
> flume-ng-core/src/main/java/org/apache/flume/client/avro/SpoolingFileLineReader.java 8362299
> flume-ng-core/src/main/java/org/apache/flume/serialization/DurablePositionTracker.java PRE-CREATION

Re: Review Request: FLUME-1632: Persist progress on each file in file spooling client/source

Mike Percy 2012-12-19, 12:41

> On Dec. 19, 2012, 1:54 a.m., Brock Noland wrote:
> > Hi Mike!
> >
> > Great patch! There are a couple issues below but otherwise I think it's ready for a commit!

Thank you so much for your time reviewing this patch, Brock! I mostly tested with the source, now I see the CLI had some issues as well.
> On Dec. 19, 2012, 1:54 a.m., Brock Noland wrote:
> > flume-ng-core/src/main/java/org/apache/flume/client/avro/AvroCLIClient.java, line 59
> > <https://reviews.apache.org/r/8596/diff/4/?file=240885#file240885line59>
> >
> > It's quite slow with such a large batch size. 4 minutes to transfer 60MB of data to a local agent (memory channel and null sink). It would be nice if that was configurable. That could be a follow up JIRA but it'd be nice to set this be a command line option.

Yes, let's do this as a follow up JIRA.
> On Dec. 19, 2012, 1:54 a.m., Brock Noland wrote:
> > flume-ng-core/src/main/java/org/apache/flume/client/avro/ReliableSpoolingFileEventReader.java, line 224
> > <https://reviews.apache.org/r/8596/diff/4/?file=240890#file240890line224>
> >
> > It's possible for currentFile to be absent.

Oops, yep.
> On Dec. 19, 2012, 1:54 a.m., Brock Noland wrote:
> > flume-ng-core/src/main/java/org/apache/flume/serialization/LineDeserializer.java, line 107
> > <https://reviews.apache.org/r/8596/diff/4/?file=240901#file240901line107>
> >
> > Seeing the error below. Guessing it's cause the last file during a run will be closed twice. One in retireCurrentFile and once in the close method. This was doubly ugly for me because it hid an exception being thrown in retireCurrentFile due to a file name violation.
> >
> >
> > java.nio.channels.ClosedChannelException
> > at sun.nio.ch.FileChannelImpl.ensureOpen(FileChannelImpl.java:88)
> > at sun.nio.ch.FileChannelImpl.position(FileChannelImpl.java:265)
> > at org.apache.flume.serialization.ResettableFileInputStream.seek(ResettableFileInputStream.java:212)
> > at org.apache.flume.serialization.ResettableFileInputStream.reset(ResettableFileInputStream.java:204)
> > at org.apache.flume.serialization.LineDeserializer.reset(LineDeserializer.java:102)
> > at org.apache.flume.serialization.LineDeserializer.close(LineDeserializer.java:107)
> > at org.apache.flume.client.avro.ReliableSpoolingFileEventReader.close(ReliableSpoolingFileEventReader.java:224)
> > at org.apache.flume.client.avro.AvroCLIClient.run(AvroCLIClient.java:217)
> > at org.apache.flume.client.avro.AvroCLIClient.main(AvroCLIClient.java:71)
> >
> >

I improved the close() handling both in the ResettableFileInputStream and in the LineDeserializer to address this issue.
- Mike
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/8596/#review14708
-----------------------------------------------------------
On Dec. 19, 2012, 12:37 p.m., Mike Percy wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/8596/
> -----------------------------------------------------------
>
> (Updated Dec. 19, 2012, 12:37 p.m.)
>
>
> Review request for Flume.
>
>
> Description
> -------
>
> Defines EventDeserializer interface and uses it from the spooling source. Progress is persisted as bytes are read from the underlying file.
>
>
> This addresses bug FLUME-1632.
> https://issues.apache.org/jira/browse/FLUME-1632
>
>
> Diffs
> -----
>
> flume-ng-core/pom.xml 0224519
> flume-ng-core/src/main/avro/TransferStateFileMeta.avsc PRE-CREATION
> flume-ng-core/src/main/java/org/apache/flume/client/avro/AvroCLIClient.java 37e9ffa
> flume-ng-core/src/main/java/org/apache/flume/client/avro/BufferedLineReader.java 718e1b2
> flume-ng-core/src/main/java/org/apache/flume/client/avro/EventReader.java PRE-CREATION
> flume-ng-core/src/main/java/org/apache/flume/client/avro/LineReader.java 904f22c

Re: Review Request: FLUME-1632: Persist progress on each file in file spooling client/source

Mike Percy 2012-12-19, 12:38

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/8596/
-----------------------------------------------------------

(Updated Dec. 19, 2012, 12:37 p.m.)
Review request for Flume.
Changes
-------

Fixed several concerns raised in the review
Description
-------

Defines EventDeserializer interface and uses it from the spooling source. Progress is persisted as bytes are read from the underlying file.
This addresses bug FLUME-1632.
   https://issues.apache.org/jira/browse/FLUME-1632
Diffs (updated)
-----

  flume-ng-core/pom.xml 0224519
  flume-ng-core/src/main/avro/TransferStateFileMeta.avsc PRE-CREATION
  flume-ng-core/src/main/java/org/apache/flume/client/avro/AvroCLIClient.java 37e9ffa
  flume-ng-core/src/main/java/org/apache/flume/client/avro/BufferedLineReader.java 718e1b2
  flume-ng-core/src/main/java/org/apache/flume/client/avro/EventReader.java PRE-CREATION
  flume-ng-core/src/main/java/org/apache/flume/client/avro/LineReader.java 904f22c
  flume-ng-core/src/main/java/org/apache/flume/client/avro/ReliableEventReader.java PRE-CREATION
  flume-ng-core/src/main/java/org/apache/flume/client/avro/ReliableSpoolingFileEventReader.java PRE-CREATION
  flume-ng-core/src/main/java/org/apache/flume/client/avro/SimpleTextLineEventReader.java PRE-CREATION
  flume-ng-core/src/main/java/org/apache/flume/client/avro/SpoolingFileLineReader.java 8362299
  flume-ng-core/src/main/java/org/apache/flume/serialization/DurablePositionTracker.java PRE-CREATION
  flume-ng-core/src/main/java/org/apache/flume/serialization/EventDeserializer.java PRE-CREATION
  flume-ng-core/src/main/java/org/apache/flume/serialization/EventDeserializerFactory.java PRE-CREATION
  flume-ng-core/src/main/java/org/apache/flume/serialization/EventDeserializerType.java PRE-CREATION
  flume-ng-core/src/main/java/org/apache/flume/serialization/EventSerDe.java PRE-CREATION
  flume-ng-core/src/main/java/org/apache/flume/serialization/EventSerializer.java a418935
  flume-ng-core/src/main/java/org/apache/flume/serialization/EventSerializerFactory.java 75853a9
  flume-ng-core/src/main/java/org/apache/flume/serialization/EventSerializerType.java afe8ed8
  flume-ng-core/src/main/java/org/apache/flume/serialization/LineDeserializer.java PRE-CREATION
  flume-ng-core/src/main/java/org/apache/flume/serialization/PositionTracker.java PRE-CREATION
  flume-ng-core/src/main/java/org/apache/flume/serialization/Resettable.java PRE-CREATION
  flume-ng-core/src/main/java/org/apache/flume/serialization/ResettableFileInputStream.java PRE-CREATION
  flume-ng-core/src/main/java/org/apache/flume/serialization/ResettableInputStream.java PRE-CREATION
  flume-ng-core/src/main/java/org/apache/flume/source/SpoolDirectorySource.java 61824d8
  flume-ng-core/src/main/java/org/apache/flume/source/SpoolDirectorySourceConfigurationConstants.java 806a661
  flume-ng-core/src/main/java/org/apache/flume/tools/PlatformDetect.java PRE-CREATION
  flume-ng-core/src/test/java/org/apache/flume/client/avro/TestBufferedLineReader.java 169abe5
  flume-ng-core/src/test/java/org/apache/flume/client/avro/TestSpoolingFileLineReader.java 740bc98
  flume-ng-core/src/test/java/org/apache/flume/serialization/ResettableTestStringInputStream.java PRE-CREATION
  flume-ng-core/src/test/java/org/apache/flume/serialization/TestDurablePositionTracker.java PRE-CREATION
  flume-ng-core/src/test/java/org/apache/flume/serialization/TestLineDeserializer.java PRE-CREATION
  flume-ng-core/src/test/java/org/apache/flume/serialization/TestReliableSpoolingFileEventReader.java PRE-CREATION
  flume-ng-core/src/test/java/org/apache/flume/serialization/TestResettableFileInputStream.java PRE-CREATION
  flume-ng-core/src/test/java/org/apache/flume/source/TestSpoolDirectorySource.java 6e87b21
  flume-ng-core/src/test/resources/TestResettableFileInputStream_1.avro PRE-CREATION
  flume-ng-core/src/test/resources/TestResettableFileInputStream_1.truncated.avro PRE-CREATION
  pom.xml b934c1d

Diff: https://reviews.apache.org/r/8596/diff/
Testing

Unit tests pass.
Thanks,

Mike Percy

Re: Review Request: FLUME-1632: Persist progress on each file in file spooling client/source

Brock Noland 2012-12-19, 15:35

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/8596/#review14720
-----------------------------------------------------------
Thanks for updating the patch! I am getting these two test failures:
Tests in error:
testGoodTrackerFile(org.apache.flume.serialization.TestDurablePositionTracker)
testPartialTrackerFile(org.apache.flume.serialization.TestDurablePositionTracker)

- Brock Noland
On Dec. 19, 2012, 12:37 p.m., Mike Percy wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/8596/
> -----------------------------------------------------------
>
> (Updated Dec. 19, 2012, 12:37 p.m.)
>
>
> Review request for Flume.
>
>
> Description
> -------
>
> Defines EventDeserializer interface and uses it from the spooling source. Progress is persisted as bytes are read from the underlying file.
>
>
> This addresses bug FLUME-1632.
> https://issues.apache.org/jira/browse/FLUME-1632
>
>
> Diffs
> -----
>
> flume-ng-core/pom.xml 0224519
> flume-ng-core/src/main/avro/TransferStateFileMeta.avsc PRE-CREATION
> flume-ng-core/src/main/java/org/apache/flume/client/avro/AvroCLIClient.java 37e9ffa
> flume-ng-core/src/main/java/org/apache/flume/client/avro/BufferedLineReader.java 718e1b2
> flume-ng-core/src/main/java/org/apache/flume/client/avro/EventReader.java PRE-CREATION
> flume-ng-core/src/main/java/org/apache/flume/client/avro/LineReader.java 904f22c
> flume-ng-core/src/main/java/org/apache/flume/client/avro/ReliableEventReader.java PRE-CREATION
> flume-ng-core/src/main/java/org/apache/flume/client/avro/ReliableSpoolingFileEventReader.java PRE-CREATION
> flume-ng-core/src/main/java/org/apache/flume/client/avro/SimpleTextLineEventReader.java PRE-CREATION
> flume-ng-core/src/main/java/org/apache/flume/client/avro/SpoolingFileLineReader.java 8362299
> flume-ng-core/src/main/java/org/apache/flume/serialization/DurablePositionTracker.java PRE-CREATION
> flume-ng-core/src/main/java/org/apache/flume/serialization/EventDeserializer.java PRE-CREATION
> flume-ng-core/src/main/java/org/apache/flume/serialization/EventDeserializerFactory.java PRE-CREATION
> flume-ng-core/src/main/java/org/apache/flume/serialization/EventDeserializerType.java PRE-CREATION
> flume-ng-core/src/main/java/org/apache/flume/serialization/EventSerDe.java PRE-CREATION
> flume-ng-core/src/main/java/org/apache/flume/serialization/EventSerializer.java a418935
> flume-ng-core/src/main/java/org/apache/flume/serialization/EventSerializerFactory.java 75853a9
> flume-ng-core/src/main/java/org/apache/flume/serialization/EventSerializerType.java afe8ed8
> flume-ng-core/src/main/java/org/apache/flume/serialization/LineDeserializer.java PRE-CREATION
> flume-ng-core/src/main/java/org/apache/flume/serialization/PositionTracker.java PRE-CREATION
> flume-ng-core/src/main/java/org/apache/flume/serialization/Resettable.java PRE-CREATION
> flume-ng-core/src/main/java/org/apache/flume/serialization/ResettableFileInputStream.java PRE-CREATION
> flume-ng-core/src/main/java/org/apache/flume/serialization/ResettableInputStream.java PRE-CREATION
> flume-ng-core/src/main/java/org/apache/flume/source/SpoolDirectorySource.java 61824d8
> flume-ng-core/src/main/java/org/apache/flume/source/SpoolDirectorySourceConfigurationConstants.java 806a661
> flume-ng-core/src/main/java/org/apache/flume/tools/PlatformDetect.java PRE-CREATION
> flume-ng-core/src/test/java/org/apache/flume/client/avro/TestBufferedLineReader.java 169abe5
> flume-ng-core/src/test/java/org/apache/flume/client/avro/TestSpoolingFileLineReader.java 740bc98
> flume-ng-core/src/test/java/org/apache/flume/serialization/ResettableTestStringInputStream.java PRE-CREATION
> flume-ng-core/src/test/java/org/apache/flume/serialization/TestDurablePositionTracker.java PRE-CREATION

Re: Review Request: FLUME-1632: Persist progress on each file in file spooling client/source

Brock Noland 2012-12-19, 17:40

> On Dec. 19, 2012, 3:35 p.m., Brock Noland wrote:
> > Thanks for updating the patch! I am getting these two test failures:
> >
> >
> > Tests in error:
> > testGoodTrackerFile(org.apache.flume.serialization.TestDurablePositionTracker)
> > testPartialTrackerFile(org.apache.flume.serialization.TestDurablePositionTracker)

This is because the included .avro files did not apply with the patch command. git apply fixes this.
- Brock
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/8596/#review14720
-----------------------------------------------------------
On Dec. 19, 2012, 12:37 p.m., Mike Percy wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/8596/
> -----------------------------------------------------------
>
> (Updated Dec. 19, 2012, 12:37 p.m.)
>
>
> Review request for Flume.
>
>
> Description
> -------
>
> Defines EventDeserializer interface and uses it from the spooling source. Progress is persisted as bytes are read from the underlying file.
>
>
> This addresses bug FLUME-1632.
> https://issues.apache.org/jira/browse/FLUME-1632
>
>
> Diffs
> -----
>
> flume-ng-core/pom.xml 0224519
> flume-ng-core/src/main/avro/TransferStateFileMeta.avsc PRE-CREATION
> flume-ng-core/src/main/java/org/apache/flume/client/avro/AvroCLIClient.java 37e9ffa
> flume-ng-core/src/main/java/org/apache/flume/client/avro/BufferedLineReader.java 718e1b2
> flume-ng-core/src/main/java/org/apache/flume/client/avro/EventReader.java PRE-CREATION
> flume-ng-core/src/main/java/org/apache/flume/client/avro/LineReader.java 904f22c
> flume-ng-core/src/main/java/org/apache/flume/client/avro/ReliableEventReader.java PRE-CREATION
> flume-ng-core/src/main/java/org/apache/flume/client/avro/ReliableSpoolingFileEventReader.java PRE-CREATION
> flume-ng-core/src/main/java/org/apache/flume/client/avro/SimpleTextLineEventReader.java PRE-CREATION
> flume-ng-core/src/main/java/org/apache/flume/client/avro/SpoolingFileLineReader.java 8362299
> flume-ng-core/src/main/java/org/apache/flume/serialization/DurablePositionTracker.java PRE-CREATION
> flume-ng-core/src/main/java/org/apache/flume/serialization/EventDeserializer.java PRE-CREATION
> flume-ng-core/src/main/java/org/apache/flume/serialization/EventDeserializerFactory.java PRE-CREATION
> flume-ng-core/src/main/java/org/apache/flume/serialization/EventDeserializerType.java PRE-CREATION
> flume-ng-core/src/main/java/org/apache/flume/serialization/EventSerDe.java PRE-CREATION
> flume-ng-core/src/main/java/org/apache/flume/serialization/EventSerializer.java a418935
> flume-ng-core/src/main/java/org/apache/flume/serialization/EventSerializerFactory.java 75853a9
> flume-ng-core/src/main/java/org/apache/flume/serialization/EventSerializerType.java afe8ed8
> flume-ng-core/src/main/java/org/apache/flume/serialization/LineDeserializer.java PRE-CREATION
> flume-ng-core/src/main/java/org/apache/flume/serialization/PositionTracker.java PRE-CREATION
> flume-ng-core/src/main/java/org/apache/flume/serialization/Resettable.java PRE-CREATION
> flume-ng-core/src/main/java/org/apache/flume/serialization/ResettableFileInputStream.java PRE-CREATION
> flume-ng-core/src/main/java/org/apache/flume/serialization/ResettableInputStream.java PRE-CREATION
> flume-ng-core/src/main/java/org/apache/flume/source/SpoolDirectorySource.java 61824d8
> flume-ng-core/src/main/java/org/apache/flume/source/SpoolDirectorySourceConfigurationConstants.java 806a661
> flume-ng-core/src/main/java/org/apache/flume/tools/PlatformDetect.java PRE-CREATION
> flume-ng-core/src/test/java/org/apache/flume/client/avro/TestBufferedLineReader.java 169abe5
> flume-ng-core/src/test/java/org/apache/flume/client/avro/TestSpoolingFileLineReader.java 740bc98
> flume-ng-core/src/test/java/org/apache/flume/serialization/ResettableTestStringInputStream.java PRE-CREATION

Re: Review Request: FLUME-1632: Persist progress on each file in file spooling client/source

Mike Percy 2012-12-20, 03:10

> On Dec. 19, 2012, 3:35 p.m., Brock Noland wrote:
> > Thanks for updating the patch! I am getting these two test failures:
> >
> >
> > Tests in error:
> > testGoodTrackerFile(org.apache.flume.serialization.TestDurablePositionTracker)
> > testPartialTrackerFile(org.apache.flume.serialization.TestDurablePositionTracker)
>
> Brock Noland wrote:
> This is because the included .avro files did not apply with the patch command. git apply fixes this.

Yes, you need the latest version of GNU patch if you need Git binary patch support in the patch utility. Thanks Brock!!
- Mike
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/8596/#review14720
-----------------------------------------------------------
On Dec. 19, 2012, 12:37 p.m., Mike Percy wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/8596/
> -----------------------------------------------------------
>
> (Updated Dec. 19, 2012, 12:37 p.m.)
>
>
> Review request for Flume.
>
>
> Description
> -------
>
> Defines EventDeserializer interface and uses it from the spooling source. Progress is persisted as bytes are read from the underlying file.
>
>
> This addresses bug FLUME-1632.
> https://issues.apache.org/jira/browse/FLUME-1632
>
>
> Diffs
> -----
>
> flume-ng-core/pom.xml 0224519
> flume-ng-core/src/main/avro/TransferStateFileMeta.avsc PRE-CREATION
> flume-ng-core/src/main/java/org/apache/flume/client/avro/AvroCLIClient.java 37e9ffa
> flume-ng-core/src/main/java/org/apache/flume/client/avro/BufferedLineReader.java 718e1b2
> flume-ng-core/src/main/java/org/apache/flume/client/avro/EventReader.java PRE-CREATION
> flume-ng-core/src/main/java/org/apache/flume/client/avro/LineReader.java 904f22c
> flume-ng-core/src/main/java/org/apache/flume/client/avro/ReliableEventReader.java PRE-CREATION
> flume-ng-core/src/main/java/org/apache/flume/client/avro/ReliableSpoolingFileEventReader.java PRE-CREATION
> flume-ng-core/src/main/java/org/apache/flume/client/avro/SimpleTextLineEventReader.java PRE-CREATION
> flume-ng-core/src/main/java/org/apache/flume/client/avro/SpoolingFileLineReader.java 8362299
> flume-ng-core/src/main/java/org/apache/flume/serialization/DurablePositionTracker.java PRE-CREATION
> flume-ng-core/src/main/java/org/apache/flume/serialization/EventDeserializer.java PRE-CREATION
> flume-ng-core/src/main/java/org/apache/flume/serialization/EventDeserializerFactory.java PRE-CREATION
> flume-ng-core/src/main/java/org/apache/flume/serialization/EventDeserializerType.java PRE-CREATION
> flume-ng-core/src/main/java/org/apache/flume/serialization/EventSerDe.java PRE-CREATION
> flume-ng-core/src/main/java/org/apache/flume/serialization/EventSerializer.java a418935
> flume-ng-core/src/main/java/org/apache/flume/serialization/EventSerializerFactory.java 75853a9
> flume-ng-core/src/main/java/org/apache/flume/serialization/EventSerializerType.java afe8ed8
> flume-ng-core/src/main/java/org/apache/flume/serialization/LineDeserializer.java PRE-CREATION
> flume-ng-core/src/main/java/org/apache/flume/serialization/PositionTracker.java PRE-CREATION
> flume-ng-core/src/main/java/org/apache/flume/serialization/Resettable.java PRE-CREATION
> flume-ng-core/src/main/java/org/apache/flume/serialization/ResettableFileInputStream.java PRE-CREATION
> flume-ng-core/src/main/java/org/apache/flume/serialization/ResettableInputStream.java PRE-CREATION
> flume-ng-core/src/main/java/org/apache/flume/source/SpoolDirectorySource.java 61824d8
> flume-ng-core/src/main/java/org/apache/flume/source/SpoolDirectorySourceConfigurationConstants.java 806a661
> flume-ng-core/src/main/java/org/apache/flume/tools/PlatformDetect.java PRE-CREATION
> flume-ng-core/src/test/java/org/apache/flume/client/avro/TestBufferedLineReader.java 169abe5

Re: Review Request: FLUME-1632: Persist progress on each file in file spooling client/source

Brock Noland 2012-12-19, 18:12

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/8596/#review14724
-----------------------------------------------------------

Ship it!
Ship It!

- Brock Noland
On Dec. 19, 2012, 12:37 p.m., Mike Percy wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/8596/
> -----------------------------------------------------------
>
> (Updated Dec. 19, 2012, 12:37 p.m.)
>
>
> Review request for Flume.
>
>
> Description
> -------
>
> Defines EventDeserializer interface and uses it from the spooling source. Progress is persisted as bytes are read from the underlying file.
>
>
> This addresses bug FLUME-1632.
> https://issues.apache.org/jira/browse/FLUME-1632
>
>
> Diffs
> -----
>
> flume-ng-core/pom.xml 0224519
> flume-ng-core/src/main/avro/TransferStateFileMeta.avsc PRE-CREATION
> flume-ng-core/src/main/java/org/apache/flume/client/avro/AvroCLIClient.java 37e9ffa
> flume-ng-core/src/main/java/org/apache/flume/client/avro/BufferedLineReader.java 718e1b2
> flume-ng-core/src/main/java/org/apache/flume/client/avro/EventReader.java PRE-CREATION
> flume-ng-core/src/main/java/org/apache/flume/client/avro/LineReader.java 904f22c
> flume-ng-core/src/main/java/org/apache/flume/client/avro/ReliableEventReader.java PRE-CREATION
> flume-ng-core/src/main/java/org/apache/flume/client/avro/ReliableSpoolingFileEventReader.java PRE-CREATION
> flume-ng-core/src/main/java/org/apache/flume/client/avro/SimpleTextLineEventReader.java PRE-CREATION
> flume-ng-core/src/main/java/org/apache/flume/client/avro/SpoolingFileLineReader.java 8362299
> flume-ng-core/src/main/java/org/apache/flume/serialization/DurablePositionTracker.java PRE-CREATION
> flume-ng-core/src/main/java/org/apache/flume/serialization/EventDeserializer.java PRE-CREATION
> flume-ng-core/src/main/java/org/apache/flume/serialization/EventDeserializerFactory.java PRE-CREATION
> flume-ng-core/src/main/java/org/apache/flume/serialization/EventDeserializerType.java PRE-CREATION
> flume-ng-core/src/main/java/org/apache/flume/serialization/EventSerDe.java PRE-CREATION
> flume-ng-core/src/main/java/org/apache/flume/serialization/EventSerializer.java a418935
> flume-ng-core/src/main/java/org/apache/flume/serialization/EventSerializerFactory.java 75853a9
> flume-ng-core/src/main/java/org/apache/flume/serialization/EventSerializerType.java afe8ed8
> flume-ng-core/src/main/java/org/apache/flume/serialization/LineDeserializer.java PRE-CREATION
> flume-ng-core/src/main/java/org/apache/flume/serialization/PositionTracker.java PRE-CREATION
> flume-ng-core/src/main/java/org/apache/flume/serialization/Resettable.java PRE-CREATION
> flume-ng-core/src/main/java/org/apache/flume/serialization/ResettableFileInputStream.java PRE-CREATION
> flume-ng-core/src/main/java/org/apache/flume/serialization/ResettableInputStream.java PRE-CREATION
> flume-ng-core/src/main/java/org/apache/flume/source/SpoolDirectorySource.java 61824d8
> flume-ng-core/src/main/java/org/apache/flume/source/SpoolDirectorySourceConfigurationConstants.java 806a661
> flume-ng-core/src/main/java/org/apache/flume/tools/PlatformDetect.java PRE-CREATION
> flume-ng-core/src/test/java/org/apache/flume/client/avro/TestBufferedLineReader.java 169abe5
> flume-ng-core/src/test/java/org/apache/flume/client/avro/TestSpoolingFileLineReader.java 740bc98
> flume-ng-core/src/test/java/org/apache/flume/serialization/ResettableTestStringInputStream.java PRE-CREATION
> flume-ng-core/src/test/java/org/apache/flume/serialization/TestDurablePositionTracker.java PRE-CREATION
> flume-ng-core/src/test/java/org/apache/flume/serialization/TestLineDeserializer.java PRE-CREATION
> flume-ng-core/src/test/java/org/apache/flume/serialization/TestReliableSpoolingFileEventReader.java PRE-CREATION

Re: Review Request: FLUME-1632: Persist progress on each file in file spooling client/source

Mike Percy 2012-12-16, 13:31

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/8596/#review14553
-----------------------------------------------------------

flume-ng-core/src/main/java/org/apache/flume/client/avro/SpoolingFileEventParser.java
<https://reviews.apache.org/r/8596/#comment30965>

   Good catch.

flume-ng-core/src/main/java/org/apache/flume/serialization/EventDeserializerFactory.java
<https://reviews.apache.org/r/8596/#comment30966>

   Good idea.

flume-ng-core/src/main/java/org/apache/flume/serialization/PositionTrackerFile.java
<https://reviews.apache.org/r/8596/#comment30967>

   sync() in this case is actually just writing a sync marker. But this sync() and flush() should be called once per Transaction.

   I am actually looking for a way to fsync() an avro data file but I have not found one yet... wonder if this is something they overlooked...
- Mike Percy
On Dec. 14, 2012, 9:56 a.m., Mike Percy wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/8596/
> -----------------------------------------------------------
>
> (Updated Dec. 14, 2012, 9:56 a.m.)
>
>
> Review request for Flume.
>
>
> Description
> -------
>
> Defines EventDeserializer interface and uses it from the spooling source. Progress is persisted as bytes are read from the underlying file.
>
>
> This addresses bug FLUME-1632.
> https://issues.apache.org/jira/browse/FLUME-1632
>
>
> Diffs
> -----
>
> flume-ng-clients/flume-ng-forwarder/pom.xml PRE-CREATION
> flume-ng-clients/flume-ng-forwarder/src/main/java/org/apache/flume/clients/forwarder/FileClient.java PRE-CREATION
> flume-ng-clients/flume-ng-forwarder/src/main/java/org/apache/flume/clients/forwarder/FlumeTool.java PRE-CREATION
> flume-ng-clients/flume-ng-forwarder/src/main/java/org/apache/flume/clients/forwarder/ToolRegistry.java PRE-CREATION
> flume-ng-clients/flume-ng-forwarder/src/main/java/org/apache/flume/clients/forwarder/ToolRunner.java PRE-CREATION
> flume-ng-clients/flume-ng-forwarder/src/test/java/org/apache/flume/clients/forwarder/Blah2.java PRE-CREATION
> flume-ng-clients/flume-ng-forwarder/src/test/resources/flume-log4jtest.properties PRE-CREATION
> flume-ng-clients/pom.xml 7ea2d1d
> flume-ng-core/pom.xml 2554a0e
> flume-ng-core/src/main/avro/TransferStateFileMeta.avsc PRE-CREATION
> flume-ng-core/src/main/java/org/apache/flume/client/avro/AvroCLIClient.java 37e9ffa
> flume-ng-core/src/main/java/org/apache/flume/client/avro/SpoolingFileEventParser.java PRE-CREATION
> flume-ng-core/src/main/java/org/apache/flume/client/avro/SpoolingFileLineReader.java 8362299
> flume-ng-core/src/main/java/org/apache/flume/serialization/EventDeserializer.java PRE-CREATION
> flume-ng-core/src/main/java/org/apache/flume/serialization/EventDeserializerFactory.java PRE-CREATION
> flume-ng-core/src/main/java/org/apache/flume/serialization/EventDeserializerType.java PRE-CREATION
> flume-ng-core/src/main/java/org/apache/flume/serialization/EventSerDe.java PRE-CREATION
> flume-ng-core/src/main/java/org/apache/flume/serialization/EventSerializer.java a418935
> flume-ng-core/src/main/java/org/apache/flume/serialization/LineDeserializer.java PRE-CREATION
> flume-ng-core/src/main/java/org/apache/flume/serialization/PositionTracker.java PRE-CREATION
> flume-ng-core/src/main/java/org/apache/flume/serialization/PositionTrackerFile.java PRE-CREATION
> flume-ng-core/src/main/java/org/apache/flume/serialization/Resettable.java PRE-CREATION
> flume-ng-core/src/main/java/org/apache/flume/serialization/ResettableFileInputStream.java PRE-CREATION
> flume-ng-core/src/main/java/org/apache/flume/serialization/ResettableInputStream.java PRE-CREATION
> flume-ng-core/src/main/java/org/apache/flume/serialization/Seekable.java PRE-CREATION