Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Hive >> mail # user >> pipeout files


+
Sam Darwin 2012-09-07, 15:28
Copy link to this message
-
Re: pipeout files
This is likely some artifact hive leaves behind.

Our filecrush tool has piece called Clean.clean

https://github.com/edwardcapriolo/filecrush

I use it to delete anything in hdfs /tmp older then N seconds.

Edward

On Fri, Sep 7, 2012 at 11:28 AM, Sam Darwin <[EMAIL PROTECTED]> wrote:
> Hi,
>
> I am seeing like one million of these files on our hadoop cluster.
>
> 1005717 files like /tmp/hdfs/hdfs_2012082902171088341605155583849.pipeout
> 1005742 files like /tmp/hdfs/hive_job_log_hdfs_201208290217_1000376604.txt
>
> My questions are:
>
> 1.   What is a .pipeout file, and can they be deleted at any time?
> What might happen if a pipeout file is removed that shouldn't be
> removed?
>
> 2.   Is it entirely up the admin to log rotate these?    Why aren't
> they rotated by default when you install the packages?
>
> Thanks,
> Sam
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB