Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Flume, mail # dev - Review Request: FLUME-1425: Create Standalone Spooling Client


+
Patrick Wendell 2012-08-04, 06:47
Copy link to this message
-
Re: Review Request: FLUME-1425: Create Standalone Spooling Client
Patrick Wendell 2012-08-04, 06:59

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/6377/
-----------------------------------------------------------

(Updated Aug. 4, 2012, 6:59 a.m.)
Review request for Flume.
Description (updated)
-------

This patch extends the existing avro client to support a file-based spooling mechanism. See in-line documentation for precise details, but the basic idea is that a user can have a spool directory where files are deposited for ingestion into flume. Once ingested the files are clearly renamed and the implementation guarantees at-least-once delivery semantics similar to those achieved within flume itself, even across failures and restarts of the JVM running the code.

I feel vaguely uneasy about building this as part of the standlone avro client rather than as its own source. An alternative would be to build this as a proper source (in fact, there are some ad-hoc transaction semantics used here which would really be a better fit for a source). Interested in hearing feedback on that as well. The benefit of having this in the avro client is that you don't need the flume runner scripts which are not windows compatible.
This addresses bug FlUME-1425.
    https://issues.apache.org/jira/browse/FlUME-1425
Diffs
-----

  flume-ng-core/src/main/java/org/apache/flume/client/avro/AvroCLIClient.java 4a5ecae
  flume-ng-core/src/main/java/org/apache/flume/client/avro/BufferedLineReader.java PRE-CREATION
  flume-ng-core/src/main/java/org/apache/flume/client/avro/LineReader.java PRE-CREATION
  flume-ng-core/src/main/java/org/apache/flume/client/avro/SpoolingFileLineReader.java PRE-CREATION
  flume-ng-core/src/test/java/org/apache/flume/client/avro/TestBufferedLineReader.java PRE-CREATION
  flume-ng-core/src/test/java/org/apache/flume/client/avro/TestSpoolingFileLineReader.java PRE-CREATION

Diff: https://reviews.apache.org/r/6377/diff/
Testing
-------

Extensive unit tests and I also built and played with this using a stub flume agent. If you look at the JIRA I have a configuration file for an agent that will print out Avro events to the command line - that's helpful when testing this.
Thanks,

Patrick Wendell

+
Jarek Cecho 2012-08-07, 07:09
+
Patrick Wendell 2012-08-07, 17:46
+
Hari Shreedharan 2012-08-07, 18:40
+
Patrick Wendell 2012-08-11, 05:32
+
Patrick Wendell 2012-08-11, 05:34
+
Patrick Wendell 2012-08-14, 21:54
+
Patrick Wendell 2012-08-14, 22:02
+
Mike Percy 2012-08-23, 10:11
+
Mike Percy 2012-10-07, 04:30
+
Patrick Wendell 2012-10-11, 18:10
+
Mike Percy 2012-10-11, 19:30
+
Patrick Wendell 2012-10-11, 18:02
+
Mike Percy 2012-10-11, 19:31
+
Patrick Wendell 2012-10-11, 19:47
+
Patrick Wendell 2012-10-12, 17:15
+
Cameron Gandevia 2012-10-15, 19:18
+
Patrick Wendell 2012-10-15, 21:11
+
Mike Percy 2012-10-15, 22:02
+
Cameron Gandevia 2012-10-16, 01:19
+
Mike Percy 2012-10-17, 00:26
+
Mike Percy 2012-10-16, 21:49
+
Patrick Wendell 2012-10-19, 18:49
+
Mike Percy 2012-10-19, 22:43
+
Mike Percy 2012-10-17, 02:21
+
Patrick Wendell 2012-10-19, 18:48
+
Patrick Wendell 2012-10-19, 17:09
+
Mike Percy 2012-10-19, 22:31
+
Brock Noland 2012-10-22, 14:26
+
Patrick Wendell 2012-10-22, 19:31
+
Patrick Wendell 2012-10-22, 20:36
+
Mike Percy 2012-11-06, 01:13
+
Mike Percy 2012-11-06, 02:12
+
Patrick Wendell 2012-11-06, 06:34
+
Alexander Alten-Lorenz 2012-11-06, 08:27
+
Alexander Alten-Lorenz 2012-11-06, 09:15
+
Mike Percy 2012-10-30, 06:22
+
Patrick Wendell 2012-11-01, 04:47