Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Flume >> mail # user >> SpoolDir marks item as completed, when sink fails


Copy link to this message
-
SpoolDir marks item as completed, when sink fails
Hello all,

I am running HDP 1.2 and Flume 1.3. I have a flume setup which includes a
(1) -  Load Balancer that uses SpoolDir adapter and sends events to Avro
sinks
(2) - Agents which consume the data using an avro source and writing to
hdfs.

During testing I noticed that there's a dissonance between the Load
Balancer and the Consumers...
When a Load Balancer process a file it marks it as COMPLETED, even if the
consumer has crashed while writing to HDFS.

A preferred behavior would be the Load Balancer to wait until the consumer
commits its transaction and reports it as successful before the file is
marked as COMPLETED. This does not allow me to verify which files has been
loaded successfully if an agent has crashed and recovery is in process.

Have I miss-configured my Agents or this is actually the desired behavior?
Kind Regards,
Tzur
+
Mike Percy 2013-02-01, 09:56
+
Tzur Turkenitz 2013-02-01, 15:44
+
Mike Percy 2013-02-05, 08:29
+
Mike Percy 2013-02-05, 08:31
+
Tzur Turkenitz 2013-02-05, 15:25