Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Flume, mail # dev - Review Request: FLUME-1425: Create Standalone Spooling Client


Copy link to this message
-
Re: Review Request: FLUME-1425: Create a SpoolDirectory Source and Client
Alexander Alten-Lorenz 2012-11-06, 09:15

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/6377/#review13136
-----------------------------------------------------------

flume-ng-core/src/test/java/org/apache/flume/client/avro/TestSpoolingFileLineReader.java
<https://reviews.apache.org/r/6377/#comment28292>

    fails:
    testBehaviorWithEmptyFile(org.apache.flume.client.avro.TestSpoolingFileLineReader)  Time elapsed: 0.005 sec  <<< FAILURE!
    java.lang.AssertionError
            at org.junit.Assert.fail(Assert.java:92)
            at org.junit.Assert.assertTrue(Assert.java:43)
            at org.junit.Assert.assertTrue(Assert.java:54)
            at org.apache.flume.client.avro.TestSpoolingFileLineReader.testBehaviorWithEmptyFile(TestSpoolingFileLineReader.java:396)

flume-ng-core/src/test/java/org/apache/flume/client/avro/TestSpoolingFileLineReader.java
<https://reviews.apache.org/r/6377/#comment28293>

    fails:
    2012-11-06 10:04:43,339 (main) [ERROR - org.apache.flume.client.avro.SpoolingFileLineReader.readLines(SpoolingFileLineReader.java:196)] Found line longer than 15 characters, cannot make progress.
    2012-11-06 10:04:43,339 (main) [ERROR - org.apache.flume.client.avro.SpoolingFileLineReader.readLines(SpoolingFileLineReader.java:200)] Invalid line starts with: reallyreallyreallyreallyreally
    2012-11-06 10:04:43,342 (main) [ERROR - org.apache.flume.client.avro.SpoolingFileLineReader.readLines(SpoolingFileLineReader.java:196)] Found line longer than 15 characters, cannot make progress.
    2012-11-06 10:04:43,342 (main) [ERROR - org.apache.flume.client.avro.SpoolingFileLineReader.readLines(SpoolingFileLineReader.java:200)] Invalid line starts with: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
    2012-11-06 10:04:43,344 (main) [ERROR - org.apache.flume.client.avro.SpoolingFileLineReader.readLines(SpoolingFileLineReader.java:196)] Found line longer than 15 characters, cannot make progress.
    2012-11-06 10:04:43,344 (main) [ERROR - org.apache.flume.client.avro.SpoolingFileLineReader.readLines(SpoolingFileLineReader.java:200)] Invalid line starts with: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
- Alexander Alten-Lorenz
On Nov. 6, 2012, 6:34 a.m., Patrick Wendell wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/6377/
> -----------------------------------------------------------
>
> (Updated Nov. 6, 2012, 6:34 a.m.)
>
>
> Review request for Flume.
>
>
> Description
> -------
>
> This patch adds a spooling directory based source. The  idea is that a user can have a spool directory where files are deposited for ingestion into flume. Once ingested, the files are clearly renamed and the implementation guarantees at-least-once delivery semantics similar to those achieved within flume itself, even across failures and restarts of the JVM running the code.
>
> This helps fill the gap for people who want a way to get reliable delivery of events into flume, but don't want to directly write their application against the flume API. They can simply drop log files off in a spooldir and let flume ingest asynchronously (using some shell scripts or other automated process).
>
> Unlike the prior iteration, this patch implements a first-class source. It also extends the avro client to support spooling in a similar manner.
>
>
> This addresses bug FlUME-1425.
>     https://issues.apache.org/jira/browse/FlUME-1425
>
>
> Diffs
> -----
>
>   flume-ng-configuration/src/main/java/org/apache/flume/conf/source/SourceConfiguration.java da804d7
>   flume-ng-configuration/src/main/java/org/apache/flume/conf/source/SourceType.java abbbf1c
>   flume-ng-core/src/main/java/org/apache/flume/client/avro/AvroCLIClient.java 4a5ecae
>   flume-ng-core/src/main/java/org/apache/flume/client/avro/BufferedLineReader.java PRE-CREATION
>   flume-ng-core/src/main/java/org/apache/flume/client/avro/LineReader.java PRE-CREATION