Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hive, mail # user - pipeout files


Copy link to this message
-
Re: pipeout files
Edward Capriolo 2012-09-07, 15:48
This is likely some artifact hive leaves behind.

Our filecrush tool has piece called Clean.clean

https://github.com/edwardcapriolo/filecrush

I use it to delete anything in hdfs /tmp older then N seconds.

Edward

On Fri, Sep 7, 2012 at 11:28 AM, Sam Darwin <[EMAIL PROTECTED]> wrote:
> Hi,
>
> I am seeing like one million of these files on our hadoop cluster.
>
> 1005717 files like /tmp/hdfs/hdfs_2012082902171088341605155583849.pipeout
> 1005742 files like /tmp/hdfs/hive_job_log_hdfs_201208290217_1000376604.txt
>
> My questions are:
>
> 1.   What is a .pipeout file, and can they be deleted at any time?
> What might happen if a pipeout file is removed that shouldn't be
> removed?
>
> 2.   Is it entirely up the admin to log rotate these?    Why aren't
> they rotated by default when you install the packages?
>
> Thanks,
> Sam