Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Pig >> mail # user >> PigServer memory leak due to calling File.deleteOnExit() for job jars.


Copy link to this message
-
PigServer memory leak due to calling File.deleteOnExit() for job jars.
Hello Pig Gurus,

I am using PigServer (
http://pig.apache.org/docs/r0.10.0/api/org/apache/pig/PigServer.html) to
schedule jobs on production (~100 per day) and realized that the cleanup of
job jar files on the local filesystem is triggered by calling
java.io.file.deleteOnExit() - indicating that these tmp files get deleted
only when the jvm shuts down gracefully. If my understanding is correct,
even if I write a separate cron to delete the leftover job jars, a long
running process/jvm which relies on using PigServer for executing pig
scripts would eventually run out of memory.

Has anyone faced such issues and if so found a work-around?

Any help/pointers are appreciated.

Other references to similar issues
grokbase.com/t/pig/user/101ts7cbzd/how-to-cleanup-old-job-jars ,
http://stackoverflow.com/questions/12711419/pigserver-leaving-job-jars-on-local-filesystem,
Thanks,
Ankit
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB