Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Flume, mail # dev - Review Request: FLUME-1632: Persist progress on each file in file spooling client/source


Copy link to this message
-
Re: Review Request: FLUME-1632: Persist progress on each file in file spooling client/source
Brock Noland 2012-12-19, 01:54

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/8596/#review14708
-----------------------------------------------------------
Hi Mike!

Great patch!  There are a couple issues below but otherwise I think it's ready for a commit!
flume-ng-core/src/main/java/org/apache/flume/client/avro/AvroCLIClient.java
<https://reviews.apache.org/r/8596/#comment31192>

    It's quite slow with such a large batch size. 4 minutes to transfer 60MB of data to a local agent (memory channel and null sink). It would be nice if that was configurable. That could be a follow up JIRA but it'd be nice to set this be a command line option.

flume-ng-core/src/main/java/org/apache/flume/client/avro/AvroCLIClient.java
<https://reviews.apache.org/r/8596/#comment31190>

    I know you didn't change this, but I hit this while testing. That exception should be logged in the logger.error().

flume-ng-core/src/main/java/org/apache/flume/client/avro/ReliableSpoolingFileEventReader.java
<https://reviews.apache.org/r/8596/#comment31189>

    It's possible for currentFile to be absent.

flume-ng-core/src/main/java/org/apache/flume/serialization/LineDeserializer.java
<https://reviews.apache.org/r/8596/#comment31191>

    Seeing the error below. Guessing it's cause the last file during a run will be closed twice. One in retireCurrentFile and once in the close method. This was doubly ugly for me because it hid an exception being thrown in retireCurrentFile due to a file name violation.
    
    
    java.nio.channels.ClosedChannelException
     at sun.nio.ch.FileChannelImpl.ensureOpen(FileChannelImpl.java:88)
     at sun.nio.ch.FileChannelImpl.position(FileChannelImpl.java:265)
     at org.apache.flume.serialization.ResettableFileInputStream.seek(ResettableFileInputStream.java:212)
     at org.apache.flume.serialization.ResettableFileInputStream.reset(ResettableFileInputStream.java:204)
     at org.apache.flume.serialization.LineDeserializer.reset(LineDeserializer.java:102)
     at org.apache.flume.serialization.LineDeserializer.close(LineDeserializer.java:107)
     at org.apache.flume.client.avro.ReliableSpoolingFileEventReader.close(ReliableSpoolingFileEventReader.java:224)
     at org.apache.flume.client.avro.AvroCLIClient.run(AvroCLIClient.java:217)
     at org.apache.flume.client.avro.AvroCLIClient.main(AvroCLIClient.java:71)
    
    
- Brock Noland
On Dec. 18, 2012, 11:21 p.m., Mike Percy wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/8596/
> -----------------------------------------------------------
>
> (Updated Dec. 18, 2012, 11:21 p.m.)
>
>
> Review request for Flume.
>
>
> Description
> -------
>
> Defines EventDeserializer interface and uses it from the spooling source. Progress is persisted as bytes are read from the underlying file.
>
>
> This addresses bug FLUME-1632.
>     https://issues.apache.org/jira/browse/FLUME-1632
>
>
> Diffs
> -----
>
>   flume-ng-core/pom.xml 0224519
>   flume-ng-core/src/main/avro/TransferStateFileMeta.avsc PRE-CREATION
>   flume-ng-core/src/main/java/org/apache/flume/client/avro/AvroCLIClient.java 37e9ffa
>   flume-ng-core/src/main/java/org/apache/flume/client/avro/BufferedLineReader.java 718e1b2
>   flume-ng-core/src/main/java/org/apache/flume/client/avro/EventReader.java PRE-CREATION
>   flume-ng-core/src/main/java/org/apache/flume/client/avro/LineReader.java 904f22c
>   flume-ng-core/src/main/java/org/apache/flume/client/avro/ReliableEventReader.java PRE-CREATION
>   flume-ng-core/src/main/java/org/apache/flume/client/avro/ReliableSpoolingFileEventReader.java PRE-CREATION
>   flume-ng-core/src/main/java/org/apache/flume/client/avro/SimpleTextLineEventReader.java PRE-CREATION
>   flume-ng-core/src/main/java/org/apache/flume/client/avro/SpoolingFileLineReader.java 8362299
>   flume-ng-core/src/main/java/org/apache/flume/serialization/DurablePositionTracker.java PRE-CREATION