Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Flume >> mail # user >> .tmp in hdfs sink


Copy link to this message
-
Re: .tmp in hdfs sink
Hi Mohit, this is a complicated issue. I've filed
https://issues.apache.org/jira/browse/FLUME-1714 to track it.

In short, it would require a non-trivial amount of work to implement this,
and it would need to be done carefully. I agree that it would be better if
Flume handled this case more gracefully than it does today. Today, Flume
assumes that you have some job that would go and clean up the .tmp files as
needed, and that you understand that they could be partially written if a
crash occurred.

Regards,
Mike

On Sun, Nov 11, 2012 at 8:32 AM, Mohit Anchlia <[EMAIL PROTECTED]>wrote:

> What we are seeing is that if flume gets killed either because of server
> failure or other reasons, it keeps around the .tmp file. Sometimes for
> whatever reasons .tmp file is not readable. Is there a way to rollover .tmp
> file more gracefully?
>