Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Flume, mail # dev - Review Request: FLUME-1632: Persist progress on each file in file spooling client/source


+
Mike Percy 2012-12-14, 09:56
+
Brock Noland 2012-12-14, 15:22
+
Mike Percy 2012-12-16, 13:31
+
Mike Percy 2012-12-16, 13:31
+
Mike Percy 2012-12-16, 13:33
+
Mike Percy 2012-12-17, 08:58
+
Brock Noland 2012-12-18, 17:44
Copy link to this message
-
Re: Review Request: FLUME-1632: Persist progress on each file in file spooling client/source
Mike Percy 2012-12-18, 21:25


> On Dec. 18, 2012, 5:44 p.m., Brock Noland wrote:
> > Looks good, have you done any manual testing of the change?

I think I found an issue with the source implementation actually, digging into it now. It processes the files but I'm seeing an issue once the directory is empty that I'm having trouble reproducing with unit tests.
> On Dec. 18, 2012, 5:44 p.m., Brock Noland wrote:
> > flume-ng-core/src/main/java/org/apache/flume/serialization/ResettableFileInputStream.java, line 98
> > <https://reviews.apache.org/r/8596/diff/3/?file=239304#file239304line98>
> >
> >     Why use direct memory here?

At least in OpenJDK, when reading from a file if you use a heap buffer it internally allocates a direct buffer then copies to the heap buffer. This is supposed to be a performance optimization.
> On Dec. 18, 2012, 5:44 p.m., Brock Noland wrote:
> > flume-ng-core/src/main/java/org/apache/flume/serialization/ResettableFileInputStream.java, line 123
> > <https://reviews.apache.org/r/8596/diff/3/?file=239304#file239304line123>
> >
> >     What is the logic here, why 4?

32 bits is the maximum size that a single character should ever take up in Unicode.
- Mike
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/8596/#review14583
-----------------------------------------------------------
On Dec. 17, 2012, 8:58 a.m., Mike Percy wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/8596/
> -----------------------------------------------------------
>
> (Updated Dec. 17, 2012, 8:58 a.m.)
>
>
> Review request for Flume.
>
>
> Description
> -------
>
> Defines EventDeserializer interface and uses it from the spooling source. Progress is persisted as bytes are read from the underlying file.
>
>
> This addresses bug FLUME-1632.
>     https://issues.apache.org/jira/browse/FLUME-1632
>
>
> Diffs
> -----
>
>   flume-ng-core/pom.xml 0224519
>   flume-ng-core/src/main/avro/TransferStateFileMeta.avsc PRE-CREATION
>   flume-ng-core/src/main/java/org/apache/flume/client/avro/AvroCLIClient.java 37e9ffa
>   flume-ng-core/src/main/java/org/apache/flume/client/avro/BufferedLineReader.java 718e1b2
>   flume-ng-core/src/main/java/org/apache/flume/client/avro/EventReader.java PRE-CREATION
>   flume-ng-core/src/main/java/org/apache/flume/client/avro/LineReader.java 904f22c
>   flume-ng-core/src/main/java/org/apache/flume/client/avro/ReliableEventReader.java PRE-CREATION
>   flume-ng-core/src/main/java/org/apache/flume/client/avro/ReliableSpoolingFileEventReader.java PRE-CREATION
>   flume-ng-core/src/main/java/org/apache/flume/client/avro/SimpleTextLineEventReader.java PRE-CREATION
>   flume-ng-core/src/main/java/org/apache/flume/client/avro/SpoolingFileLineReader.java 8362299
>   flume-ng-core/src/main/java/org/apache/flume/serialization/DurablePositionTracker.java PRE-CREATION
>   flume-ng-core/src/main/java/org/apache/flume/serialization/EventDeserializer.java PRE-CREATION
>   flume-ng-core/src/main/java/org/apache/flume/serialization/EventDeserializerFactory.java PRE-CREATION
>   flume-ng-core/src/main/java/org/apache/flume/serialization/EventDeserializerType.java PRE-CREATION
>   flume-ng-core/src/main/java/org/apache/flume/serialization/EventSerDe.java PRE-CREATION
>   flume-ng-core/src/main/java/org/apache/flume/serialization/EventSerializer.java a418935
>   flume-ng-core/src/main/java/org/apache/flume/serialization/EventSerializerFactory.java 75853a9
>   flume-ng-core/src/main/java/org/apache/flume/serialization/EventSerializerType.java afe8ed8
>   flume-ng-core/src/main/java/org/apache/flume/serialization/LineDeserializer.java PRE-CREATION
>   flume-ng-core/src/main/java/org/apache/flume/serialization/PositionTracker.java PRE-CREATION
>   flume-ng-core/src/main/java/org/apache/flume/serialization/Resettable.java PRE-CREATION
+
Brock Noland 2012-12-18, 21:29
+
Mike Percy 2012-12-18, 21:50
+
Mike Percy 2012-12-18, 23:22
+
Brock Noland 2012-12-18, 23:42
+
Brock Noland 2012-12-19, 00:00
+
Brock Noland 2012-12-19, 01:54
+
Mike Percy 2012-12-19, 12:41
+
Mike Percy 2012-12-19, 12:38
+
Brock Noland 2012-12-19, 15:35
+
Brock Noland 2012-12-19, 17:40
+
Mike Percy 2012-12-20, 03:10
+
Brock Noland 2012-12-19, 18:12