Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Flume >> mail # user >> At flume shutdown only one HDFSSink is closed properly


Copy link to this message
-
Re: At flume shutdown only one HDFSSink is closed properly
On Mon, Aug 27, 2012 at 7:53 AM, Bhaskar V. Karambelkar <[EMAIL PROTECTED]
> wrote:

> Hi Mike,
> There are indeed Jira issues for this.
> A flume jira issue , which was fixed in 1.2.0.  FLUME-1163.
>

Thanks Bhaskar, I just marked that a dup of
https://issues.apache.org/jira/browse/FLUME-1219 which has the relevant
patch.

But that is only half the story, the hadoop jira issue for supporting the
> 'fs.automatic.shutdown' flag
> is logged at
> https://issues.apache.org/jira/browse/HADOOP-4829
>
> There is even a mail thread about it on this mailing list
>
> https://mail-archives.apache.org/mod_mbox/flume-user/201208.mbox/%3CCAFukC%3D5fTnQTofaBpXUFPd%2Bqr%3DjU1G7iEpnwPX3VVtZ3Kg6VZA%40mail.gmail.com%3E
>
> Hadoop 1.x don't have the 4829 patch, so we can't use Hadoop 1.x with
> Flume if we want proper
> renaming of .tmp files at flume shutdown.
>
> CDH4, as well as CDH3u5 do have that patch pulled in, so they are fine.
>

Bummer, I did not realize that Hadoop 1.0.3 did not have the relevant
functionality. It would be nice to get that patch backported. If there is
another way around this, I'm open to suggestions.

Regards,
Mike
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB