Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Flume >> mail # user >> At flume shutdown only one HDFSSink is closed properly


+
Bhaskar V. Karambelkar 2012-08-23, 22:24
+
Mike Percy 2012-08-24, 01:35
+
Bhaskar V. Karambelkar 2012-08-24, 19:13
+
Mike Percy 2012-08-24, 19:40
+
Bhaskar V. Karambelkar 2012-08-27, 14:53
Copy link to this message
-
Re: At flume shutdown only one HDFSSink is closed properly
On Mon, Aug 27, 2012 at 7:53 AM, Bhaskar V. Karambelkar <[EMAIL PROTECTED]
> wrote:

> Hi Mike,
> There are indeed Jira issues for this.
> A flume jira issue , which was fixed in 1.2.0.  FLUME-1163.
>

Thanks Bhaskar, I just marked that a dup of
https://issues.apache.org/jira/browse/FLUME-1219 which has the relevant
patch.

But that is only half the story, the hadoop jira issue for supporting the
> 'fs.automatic.shutdown' flag
> is logged at
> https://issues.apache.org/jira/browse/HADOOP-4829
>
> There is even a mail thread about it on this mailing list
>
> https://mail-archives.apache.org/mod_mbox/flume-user/201208.mbox/%3CCAFukC%3D5fTnQTofaBpXUFPd%2Bqr%3DjU1G7iEpnwPX3VVtZ3Kg6VZA%40mail.gmail.com%3E
>
> Hadoop 1.x don't have the 4829 patch, so we can't use Hadoop 1.x with
> Flume if we want proper
> renaming of .tmp files at flume shutdown.
>
> CDH4, as well as CDH3u5 do have that patch pulled in, so they are fine.
>

Bummer, I did not realize that Hadoop 1.0.3 did not have the relevant
functionality. It would be nice to get that patch backported. If there is
another way around this, I'm open to suggestions.

Regards,
Mike